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UCAVs  are  advanced  weapon  systems  that  can  loiter  autonomously  in  a 
pack  over  a  target  area,  detect  and  acquire  the  targets,  and  then  attack  them. 
Modeling  these  capabilities  in  a  specific  hostile  operational  setting  is  necessary 
for  addressing  weapons’  design  and  operational  issues.  While  much  attention 
has  been  given  to  the  engineering  and  technological  aspects  of  UCAV 
developments,  there  are  very  few  studies  on  operational  concepts  for  these 
weapon  systems  and  their  effectiveness  and  efficiency.  This  thesis  builds 
probability  models  (Markov  Chains)  that  describe  UCAV  operations,  defines 
Measures  of  Effectiveness  (MOEs)  for  the  engagement  performance,  maps  the 
functional  relations  between  the  parameters  and  the  MOEs,  and  obtains  insights 
regarding  the  design  of  the  UCAVs  and  their  tactical  employment.  The  models 
are  used  to  conduct  extensive  numerical  analysis,  based  on  experimental  design 
concepts  and  traditional  sensitivity  analysis.  The  main  focus  of  the  analysis  is  to 
investigate  optimal  and  robust  mixes  of  UCAVs  of  different  types,  with  respect  to 
the  MOEs.  While  in  most  cases,  extreme-point  solutions  are  optimal,  there  are 
cases  where  a  balanced  UCAV  mix  is  better. 
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I.  INTRODUCTION 


A.  FOREWORD 

In  this  chapter  we  present  some  aspects  of  Unmanned  Aerial  Vehicles 
(UAVs)  -  in  particular  Unmanned  Combat  Aerial  Vehicles  (UCAVs)  -  and 
describe  the  problem  addressed  in  this  thesis.  We  also  discuss  the  methods  and 
techniques  used  for  solving  the  problem  and  for  making  useful  inference  from  the 
results.  Finally,  we  outline  the  thesis  chapters. 

B.  UNMANNED  (COMBAT)  AERIAL  VEHICLES 

UAVs  are  mobile  airborne  machines  that  do  not  require  an  on-board 
human  operator.  Typically  they  are  controlled  by  a  remote  operator  or 
autonomous  control  logic  (Corner  and  Lamont,  2004). 

The  Department  of  Defense  (DoD)  defines  UAVs  as  “powered,  aerial 
vehicles  that  do  not  carry  a  human  operator,  use  aerodynamic  forces  to  provide 
vehicle  lift,  can  fly  autonomously  or  be  piloted  remotely,  can  be  expendable  or 
recoverable,  and  can  carry  a  lethal  or  non-lethal  payload.”  (Bone  and  Bolkcom, 
2003). 

UAVs  are  a  critical  part  of  (future)  armed  forces,  that  consists  of  highly 
mobile  and  network  enabled  systems  with  integrated  sensors  and  precision 
munitions.  UAVs  either  provide  eyes  on  the  battlefield  that  trigger  the 
deployment  of  precision  munitions  by  other  platforms,  or  engage  targets 
themselves  (UCAVs)  (Sulewski,  2005).  In  addition  to  triggering  the  deployment 
of  precision  munitions,  and  providing  situational  awareness  of  the  engagement 
area,  UAVs  assist  in  all  communication  aspects  throughout  the  theater  of 
operations. 

The  increasingly  important  role  of  UAVs  in  warfare  is  demonstrated  by  the 
U.S.  Army’s  resolution  to  have  these  systems  at  the  core  of  its  FCS  FoS  (Future 
Combat  Systems  Family  of  Systems).  FCS  UAVs  are  broken  down  into  four 
classes  according  to  their  capabilities.  Class  I  UAVs  provide  RSTA 
(Reconnaissance,  Surveillance,  Target  Acquisition)  capabilities  at  the  platoon 
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level.  Class  II  UAVs  provide  RSTA  capabilities  and  target  designation  at  the 
platoon  and  company  level.  Class  III  UAVs  provide  RSTA  capabilities,  target 
designation,  communication  relay,  and  mine  detection  at  the  CAB  (Combined 
Arms  Battalion)  level,  while  class  IV  UAVs  provide  similar  capabilities  at  the  UA 
(Unit  of  Action)  level  (Sulewski,  2005). 

The  future  military  force  will  be  complex:  a  highly  integrated  mix  of 
manned  and  unmanned  units.  These  unmanned  units  could  function  individually 
or  within  a  swarm.  The  addition  of  unmanned  units  will  decrease  the  danger  that 
soldiers  face  in  direct  combat.  The  tendency  is  to  either  have  a  single  operator 
controlling  a  swarm  of  UAVs,  or  to  let  them  operate  autonomously  with  no  human 
supervision.  The  ability  to  use  autonomous  vehicles  to  perform  wartime  mission 
is  an  important  application  in  future  military  operations.  Technology  in  the  UAV 
arena  is  also  moving  toward  smaller  and  more  capable  systems. 

One  of  the  initial  motivations  that  served  as  impetus  for  developing  UAVs 
was  that  UAVs  would  be  inexpensive.  They  could  be  launched  into  high  risk 
missions  without  risking  a  costly  manned  aircraft  and  the  lives  of  its  crew.  Of 
course  as  the  UAVs  continuously  grow  in  complexity  and  utility,  they  also 
increase  in  cost,  and  therefore  it  becomes  more  crucial  for  them  to  be  highly 
combat  effective  (McMindes,  2005).  The  effectiveness  of  UAVs  in  battle 
depends  on  many  factors,  some  of  which  are  addressed  in  this  thesis.  Exploring 
these  factors  may  let  us  better  understand  what  design  characteristics  or 
operational  decisions  would  lead  to  a  more  effective  (and  cost-efficient)  use  of 
UAVs. 

The  use  of  UCAVs  removes  the  risk  of  aircrew  being  killed,  injured  or 
captured  if  the  vehicle  is  shot  down  or  lost  due  to  mechanical  failure.  Airframe 
designs  can  be  smaller  and  lighter  than  their  manned  counterparts  and  can  be 
designed  for  longer  endurance.  Also,  UCAV  platforms  are  cheaper  to  buy  and 
operate,  and  require  less  expensive  testing  and  training.  These  might  be  among 
the  main  advantages  in  future  planning  (Baggesen,  2005). 

Modern  UCAVs  are  navigated  and  guided  by  radar,  video,  infrared 
cameras,  lasers,  and  Inertial  Navigation  Systems  (INS)  and  aided  by  the  satellite 
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based  Global  Positioning  System  (GPS).  The  enhancement  of  sensor  systems, 
processor  units,  decision  making  algorithms,  and  terminal  seekers  leads  to 
autonomy  for  target  acquisition,  recognition,  and  attack.  These  capabilities, 
combined  with  inexpensive  designs  and  operational  opportunities,  make  UCAVs 
a  disruptive  technology  on  the  battlefield.  UCAVs  enable  war  fighters  to  attack 
targets  with  weapon  systems  that  can  operate  in  highly  defended  areas,  and 
cause  less  collateral  damage,  due  to  enhanced  precision.  There  no  longer 
seems  to  be  a  trade-off  between  own  casualties  and  the  effect  of  attacks.  This  is 
especially  important  for  a  society  that  is  perceived  as  being  less  and  less  tolerant 
of  high-casualty  engagements  and  collateral  damage  (Baggesen  2005). 

Experts  worry  that  the  more  abstract  the  use  of  weapon  systems 
becomes,  the  more  abstract  the  enemy  becomes,  and  as  humans  recede  from 
the  battlefield  as  combatants,  war  will  become  more  likely,  not  less. 

Autonomous  technology  is  still  not  completely  unleashed.  For  the  near 
future,  however,  UCAV  developers  believe  that  the  man-in-the-loop  will  be  the 
weakest  part  of  the  weapon  system  because  humans  will  be  too  slow  for  the 
decision-making  cycle,  causing  underperformance  and  collateral  damage 
(Baggesen  2005). 


C.  PROBLEM  STATEMENT 

UCAVs  are  advanced  weapon  systems  that  can  loiter  autonomously  in  a 
pack  over  a  target  area,  detect  and  acquire  the  targets,  and  then  engage  them. 
Modeling  these  capabilities  in  a  specific  hostile  operational  setting  is  necessary 
for  addressing  weapons’  design  and  operational  issues. 

While  much  attention  is  given  to  the  engineering  and  technological 
aspects  of  UCAV  developments,  there  are  very  few  studies  on  operational 
concepts  for  these  weapon  systems  and  their  effectiveness  and  efficiency.  The 
wide  range  of  design  and  operational  factors  and  capabilities  of  such 
autonomously  acting  and  interacting  weapons  will  most  likely  lead  to  a  wide 
range  of  engagement  performance  in  various  scenarios. 
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In  the  present  thesis  we  consider  a  combat  situation  involving  two  types  of 
UCAVs  against  two  types  of  passive  ground  targets  and  we  seek  answers  to  the 
following  issues: 

•  The  effect  of  UCAV  design  and  operational  parameters  on  the  end  state  of 
the  engagement  in  the  presence  of  imperfect  situational  awareness. 

•  Suitable  Measures  Of  Effectiveness  (MOEs)  for  measuring  the  effect  of 
UCAV  design  and  tactics. 

•  Sensitivity  of  the  values  of  the  MOE’s  to  the  UCAV  parameters. 

•  Given  the  capabilities  of  the  various  types  of  UCAVs,  the  best  mix  of  these 
UCAV’s  that  optimizes  certain  operational  goals. 

D.  TECHNIQUES  AND  METHODS 

In  order  to  gain  insight  about  real  combat  situations,  we  need  to  first 
model  it  using  appropriate  techniques  and  then  analyze  it  in  the  hope  of  deriving 
useful  conclusions  that  will  help  decision  makers  make  better  choices  when  it 
comes  to  selecting  parameter  settings  of  the  weapon  systems  or  developing 
scenarios  and  deployment  tactics. 

For  modeling  the  combat  situation  addressed  in  this  thesis,  a  continuous 
time  Markov  chain  (CTMC)  model  is  developed.  This  model  represents  all  the 
UCAV  design  and  environmental  parameters  and  also  contains  temporal 
information  (i.e.,  the  expected  duration  of  a  UCAV  operation).  The  basic  idea 
behind  the  CTMC  is  that  at  each  moment  in  time  the  system  can  be  described  by 
the  state  it  is  in.  The  state,  in  general,  contains  information  about  the  number  of 
UCAVs  and  targets  alive.  All  of  the  parameter  information  is  kept  in  the  model 
and  is  used  in  order  to  calculate  the  next  state  and  the  elapsed  time.  The 
outcomes  resulting  from  the  model  are  used  to  address  the  issues  described 
above  in  Section  C.  In  this  thesis,  two  MOEs  are  calculated,  the  expected 
relative  effectiveness  and  the  expected  time  of  the  operation,  both  described  in 
Chapter  III.  The  parameter  value  settings  along  with  their  respective  MOE 
outputs  are  then  analyzed  according  to  two  techniques.  The  first  one  is  the 
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NOLH  DOE  (Nearly  Orthogonal  Latin  Hypercube  Design  of  Experiments).  This 
method  allows  for  an  exploration  as  broad  as  the  analyst  deems  necessary,  and 
also  allows  for  exposing  potential  interactions  among  the  various  factors.  One 
product  of  this  type  of  analysis  is  a  regression  equation  that  can  be  used  for  its 
explanatory  power  but  also  as  a  quick  substitute  for  the  Markov  model.  The 
second  technique  is  the  traditional  sensitivity  analysis  that  helps  magnify  the 
effects  of  particular  factors  when  everything  else  is  kept  constant.  These  two 
techniques  complement  each  other  and  provide  a  comprehensive  view  of  the 
combat  situation  we  are  modeling.  The  statistical  package  JMP  is  used  for  most 
of  the  DOE  and  analysis  part,  and  Excel  is  used  for  generating  and  formatting  the 
various  plots  during  the  analysis. 

E.  THESIS  FLOW 

In  the  next  chapter  we  discuss  previous  UCAV  related  models  that 
employed  different  approaches  and  conveyed  certain  takeaways.  Also,  we 
expand  on  the  Markov  chain  and  DOE  concepts.  Chapter  III  describes  the 
combat  situation  and  develops  the  basic  Markov  model.  It  also  gives  a  thorough 
discussion  of  the  analysis  performed  using  various  methods,  as  well  as 
conclusions  and  recommendations.  Supporting  documentation  on  the 
programming  code  is  contained  in  the  Appendix. 
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II.  BACKGROUND  -  LITERATURE  REVIEW 


In  this  chapter  we  discuss  previous  Operations  Research  (OR)  models 
related  to  UCAVs,  continuous  time  Markov  chains,  and  basic  concepts  of 
experimental  design  related  to  the  analysis  presented  later  on. 

A.  PREVIOUS  UCAV  MODELS 

Progress  on  the  various  technologies  of  UCAVs  is  promising,  but  there  are 
insufficient  analytical  tools  for  evaluating  the  effectiveness  of  these  weapon 
systems  in  operational  settings.  Most  of  the  research  on  modeling  UCAV 
operations  relies  on  simulation  and  not  on  analytic  modeling.  In  (Jacques,  2002) 
the  author  presents  some  basic  analytic  results  on  the  single  UCAV/single  target 
and  general  multi-UCAV/multi-target  cases.  It  is  shown  there,  that  analytically  it 
becomes  intractable  to  develop  a  mathematical  formulation  for  arbitrary  numbers 
of  munitions  executing  arbitrarily  specified  search  patterns.  These  are  the  cases, 
however,  that  are  most  interesting  operationally,  and  the  most  practical  way  of 
performing  this  more  general  analysis  is  by  a  numerical  simulation. 

Some  work  has  been  done  on  investigating  the  possibility  of  having 
UCAVs  share  information  and  act  in  a  cooperative  fashion.  Cooperative 
behavior  is  being  investigated  to  improve  the  overall  mission  effectiveness.  The 
general  problem  addressed  is  typically  how  to  best  find  and  engage  an  unknown 
number  of  targets  in  unknown  locations  using  multiple  UCAVs.  In  (Frelinger,  et 
al,  1998)  it  is  stated  that  while  an  individual  UCAV  may  be  less  capable  than 
conventional  munitions,  through  communication  across  the  swarm  of  weapons, 
the  group  may  exhibit  behaviors  and  capabilities  that  can  exceed  those  of  more 
conventional  systems  that  do  not  employ  communication  between  weapons.  The 
potential  benefits,  which  come  about  through  shared  knowledge  include  relaxed 
sensor  performance  requirements,  robustness  to  increases  in  target  location 
errors,  and  adaptivity  to  attrition  and  poor  target  characterization.  In  (Gillen  and 
Jacques,  2002)  an  attempt  is  made  to  emulate  the  behavior  of  UCAVs  via 
simulation,  and  measure  their  overall  expected  performance.  One  extension  to 
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the  approach  described  in  (Frelinger,  et  al,  1998)  is  taking  into  account  the 
degradation  due  to  false  target  attacks.  The  simulation  allows  for  any  number  of 
targets  with  varying  priority  levels,  as  well  as  non-targets  (military  or  civilian),  and 
it  is  very  flexible  in  its  capabilities  to  handle  a  multitude  of  input  parameters  and 
supply  multiple  outputs,  such  as  total  hits  or  total  kills.  During  this  simulation,  the 
UCAVs  employ  a  decision  algorithm.  It  is  shown  that  the  selection  of  the  optimal 
weights  of  the  factors  in  this  decision  algorithm  are  very  sensitive  to  almost  all 
battlefield  characteristics,  therefore  producing  no  robust  conclusions,  apart  from 
the  fact  that  cooperative  engagement  alone  is  not  able  to  compensate  for  higher 
false  target  attack  rates. 

In  (Kress  et  al,  2006),  several  analytic  probability  models,  which  range 
from  a  simple  regenerative  formula  to  a  large-scale  continuous-time  Markov 
chain,  are  developed,  with  the  objective  to  address  design  and  operational  issues 
of  UCAVs  operating  in  hostile  environments.  The  focus  is  on  autonomous 
UCAVs,  which  are  designed  to  operate  as  a  pack  of  vehicles  that  autonomously 
search,  detect,  acquire  and  attack  targets.  The  main  idea  is  that  while  target 
detection  and  recognition  capabilities,  and  weapon  accuracy  and  lethality 
determine  the  effectiveness  of  a  single  UCAV,  two  phenomena  may  affect  the 
performance  of  the  UCAVs  as  a  pack  -  multiple  acquisitions  and  multiple  kills. 
Also,  the  impact  of  memory  on  the  acquisition  capabilities  of  a  UCAV  is  studied  in 
this  paper.  It  is  shown  that,  under  reasonable  assumptions,  memory  is  a  rather 
redundant  design  feature  in  UCAVs,  unless  we  consider  time-critical  missions. 
Some  other  takeaways  are  that  detection  rate  is  a  major  factor  in  determining  the 
operation  length,  that  attack  coordination  among  UCAVs  is  not  significant  (at 
least  for  certain  examined  scenarios),  and  that  UCAV  sensor  specificity  is  more 
important  than  sensitivity.  The  models  described  in  the  paper  above  are  limited 
to  homogeneous  targets  and  homogeneous  UCAVs.  The  present  thesis  extends 
these  concepts  to  multi-type  UCAVs  and  non-homogeneous  targets. 
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B.  CONTINUOUS  TIME  MARKOV  CHAINS 

A  Continuous  Time  Markov  Chain  (CTMC)  is  used  for  modeling  a 
probabilistically  evolving  situation  where  time  is  continuous  and  the  time  periods 
between  changes  in  the  situation  follow  an  exponential  distribution.  The  situation 
is  fully  described  by  the  state  it  is  in.  Each  state  can  transition  to  another  feasible 
state.  The  selection  of  the  next  state  is  the  result  of  an  ‘exponential  race’,  that  is 
the  next  state  is  determined  by  the  event  that  happens  first,  where  the  time  until 
the  next  event  follows  an  exponential  distribution.  Each  state  can  be  either 
transient  or  absorbing.  The  model  keeps  on  running  as  long  as  the  visited  states 
are  transient.  When  an  absorbing  state  is  reached,  the  process  stops.  The  core 
of  our  UCAV  model  is  the  CTMC  ‘engine’,  around  which  other  functions  can  be 
built.  The  CTMC  model  calculates  the  values  of  the  various  MOEs,  as  well  as 
the  expected  duration  of  the  process.  The  underlying  assumption,  though,  is  that 
the  rates  of  events  follow  a  Poisson  distribution.  This  is  not  a  long  stretch,  if  we 
consider  that  the  behavior  of  many  actual  systems  approximately  follows  this 
distribution.  The  exponential  assumption,  and  the  consequent  ability  to  use  a 
CTMC  gives  the  analyst  an  enormous  analytical  and  computational  advantage 
since  the  math  would  be  very  difficult  or  intractable  otherwise. 

C.  ASPECTS  OF  EXPERIMENTAL  DESIGNS  AND  DATA  ANALYSIS 

In  this  thesis  we  implement  two  types  of  model  analysis:  an  experimental 
design  and  a  single-factor  sensitivity  analysis  with  respect  to  a  base  case.  A 
good  Design  of  Experiments  (DOE)  allows  for  simultaneously  assessing  the 
impact  of  more  than  one  factor,  and  identifying  potential  interactions  among  the 
factors.  On  the  other  hand,  single-factor  sensitivity  analysis,  when  all  other 
factors  remain  fixed  at  their  base-case  values,  may  better  reveal  the  effect  of 
certain  parameters  in  the  neighborhood  of  a  realistic  base  case. 

The  primary  objectives  of  computer  experiments,  according  to  (Sacks  et 
al,  1989)  are:  predicting  the  response  at  untried  inputs,  optimizing  a  function  of 
the  input  factors,  or  calibrating  the  computer  code  to  physical  data.  A  more 
modern  approach  (Sanchez,  2001)  contends  that  the  appropriate  objectives 
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should  be:  developing  a  basic  understanding  of  a  particular  system,  finding 
robust  decisions,  tactics,  or  strategies,  or  comparing  the  merits  of  various 
decisions. 

There  are  several  DOE  structures  to  choose  from.  In  this  thesis  we 
employ  Latin  Hypercubes  and  more  specifically  Nearly-Orthogonal  Latin 
Hypercubes  (NOLH)  for  various  reasons  discussed  in  the  next  section. 

1.  Latin  Hypercubes 

The  challenge  in  conducting  analysis  is  in  the  curse  of  dimensionality.  In 
general  we  need  L  x  F  design  points  where  F  is  the  number  of  factors  and  L  is 
the  number  of  levels  of  each  factor,  in  order  to  cover  all  the  possible 
combinations.  This  is  known  as  a  full  factorial  design.  As  we  raise  the  number  of 
factors  and  desired  levels  to  accommodate  the  idea  of  data  farming  the  number 
of  design  points  quickly  gets  out  of  hand. 

A  NOLH  DOE  addresses  how  to  sample  the  design  space  without  looking 
at  all  possible  combinations.  It  is  beyond  the  scope  of  this  thesis  to  explain  in 
detail  how  and  why  this  works,  but  we  can  imagine  the  NOLH  DOE  as  selecting 
interior  points  from  the  parameter  space  additionally  to  the  corner  points  that  a 
factorial  would  select.  Those  interior  points  are  selected  such  that  the  correlation 
between  factor  levels  is  very  low,  so  that  we  get  a  much  more  complete  picture  of 
the  landscape  from  which  we  are  sampling.  The  low  correlation  and  the  large 
number  of  design  points  allow  the  analysis  of  both  main  effects  and  interactions 
between  factors  without  sampling  at  all  combinations  of  levels  of  each  factor.  By 
the  application  of  data  farming  and  NOLH  a  very  broad  parameter  space  can  be 
explored  and  robust  solutions  can  be  found.  A  robust  solution  may  not  be  the 
optimal  choice  for  any  given  set  of  parameters,  but  is  a  good  overall  choice  given 
a  variety  of  possibilities  (McMindes  2005  and  Cioppa  2002) . 

NOLH  is  a  very  good  all-purpose  design,  particularly  when  all  or  most  of 
the  factors  are  quantitative.  It  is  apparently  efficient,  it  has  excellent  space-filling 
properties,  and  it  adds  flexibility  by  imposing  fewer  restrictions  on  the  number  of 
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the  factors  and  their  levels.  Also,  it  allows  us  to  fit  many  different  types  of 
complex  metamodels  to  multiple  MOEs. 
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III.  THE  BASIC  MODEL 


A.  THE  COMBAT  SITUATION  AND  ITS  MODELING  OBJECTIVES 

1 .  Detailed  Description  of  the  Combat  Situation 

The  basic  combat  situation  modeled  in  this  thesis  is  that  of  a  swarm  of 
UCAVs  loitering  over  an  area  of  interest,  looking  for  operationally  valuable 
ground  targets,  acquiring  them,  and  finally  attacking  them. 

There  are  two  types  of  UCAVs,  A  and  B,  and  two  types  of  Valuable 
Targets  (VTs),  1  and  2,  in  the  target  area.  Also,  there  are  other  types  of  targets 
of  no  operational  value  called  type-3  targets.  The  latter  targets,  along  with  all 
killed  VTs  of  any  type,  are  collectively  referred  to  as  Non-Valuable  Targets 
(NVTs). 

The  type-1  and  type-2  targets  can  be  anything  valuable  to  the  enemy;  for 
example  type-1  targets  could  be  C2  (Command  and  Control)  vehicles,  and  type-2 
targets  could  be  soft-skin  military  trucks.  A  mix  of  UCAVs  of  both  types  A  and  B, 
attack  the  target  area. 

Throughout  we  assume  that  the  total  number  of  attacking  UCAVs  is  16 
which  typically  corresponds  to  two  squadrons.  The  number  of  targets,  of  both 
types,  varies  according  to  the  scenario  that  is  considered.  Nonetheless  the 
number  of  targets  of  each  type  is  confined  within  a  certain  range. 

Each  UCAV  is  disposable  (as  opposed  to  retrievable),  meaning  that  the 
weapon  is  an  integral  part  of  the  aerial  vehicle;  the  UCAV  searches,  detects,  and 
acquires  a  target,  and  finally  attacks  it.  Also,  once  launched,  a  UCAV  cannot 
return  to  base,  so  eventually,  if  not  used,  it  is  wasted. 

The  UCAVs  loiter  over  the  target  area  and  search  for  targets 
independently,  in  a  random  search  pattern.  We  assume  that  a  UCAV  has  no 
memory  of  any  previous  detection,  but  it  has  the  capability  to  recognize  an  NVT, 
although  this  capability  is  imperfect;  there  is  some  probability  for  erroneous 
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classifications.  Also,  there  is  continuous  communication  among  the  UCAVs  that 
leads  to  perfect  coordination,  in  the  sense  that  simultaneous  multiple  attacks  on 
a  single  target  do  not  occur. 

The  targets  are  passive  in  the  sense  that  they  do  not  fight  back.  So,  there 
is  no  UCAV  attrition  due  to  fire  by  the  targets.  The  combat  situation  we  are 
considering  has  a  limited  time  length  which  is  in  all  cases  small  enough 
compared  to  the  fuel  capacity  of  the  UCAVs.  Therefore,  the  UCAVs  never  run 
the  risk  of  being  wasted  due  to  fuel  shortage.  Because  there  is  no  reinforcement 
of  UCAVs,  this  combat  situation  reaches  an  absorbing  state,  when  all  the  UCAVs 
have  been  disposed. 

In  summary,  each  UCAV  loiters  above  the  area  of  interest,  until  it  detects 
a  target  which  it  tries  to  identify.  If  the  target  is  identified  as  a  NVT,  the  UCAV 
takes  no  action  and  keeps  on  loitering.  If  the  target  is  identified  as  a  VT,  the 
UCAV  promptly  acquires  and  attacks  it.  The  identification  may  or  may  not  be 
correct.  The  attack  may  or  may  not  be  successful.  In  addition  to  unnecessary  or 
unsuccessful  attacks,  a  UCAV  can  also  be  wasted  due  to  a  mechanical  failure. 

2.  The  Modeling  Objectives  and  MOEs 
a.  Objectives 

The  combat  situation  involves  several  parameters  that  describe  the 
UCAVs’  performance  (e.g.,  kill  probabilities),  and  the  operational  environment 
(e.g.  the  number  of  targets  in  the  area  of  interest).  The  values  of  these 
parameters  are  not  always  known  accurately  and  with  certainty.  For  example, 
the  mission  planners  cannot  be  absolutely  positive  about  the  kill  probabilities  of  a 
type-B  UCAV,  but  they  may  have  an  idea  about  the  range  of  possible  values  for 
those  parameters.  In  the  same  manner,  the  planners  cannot  accurately  predict 
the  number  of  type-1  targets  the  enemy  will  employ,  but  they  might  well  be  able 
to  provide  a  valid  range  for  it.  These  issues  are  discussed  in  further  detail  in  the 
experimental  design  sections  later  on. 

The  objective  is  to  build  a  stochastic  model  of  the  combat  situation 
described  above,  and  implement  it  in  order  to  gain  insights  and  produce  useful 
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takeaways  on  design  aspects  of  the  UCAVs  as  well  as  on  operational  issues. 
We  use  two  different  approaches  for  the  analysis.  The  first  approach  is  by 
means  of  a  Design  Of  Experiments  (DOE),  and  the  second  one  is  by  defining  a 
reasonable  base  case  for  all  the  parameter  values  and  performing  sensitivity 
analysis  for  each  parameter  separately. 

The  main  focus  of  the  analysis  is  to  investigate  optimal  UCAV 
mixes  with  respect  to  appropriate  Measures  Of  Effectiveness  (MOEs)  defined 
below,  and  under  uncertainty  regarding  the  values  of  the  parameters. 

b.  Measures  of  Effectiveness 

In  order  to  measure  the  mission  success,  we  define  the  following 

MOEs: 

•  Expected  value  of  killed  targets 

Assuming  that  type-1  and  type-2  targets  have  operational  values  assigned 
to  them,  this  is  the  total  accumulated  expected  value  of  all  the  killed 
targets. 

•  Expected  relative  effectiveness 

This  is  the  ratio  of  the  expected  total  number  of  killed  VTs,  over  the  initial 
total  number  of  VTs. 

•  Probability  to  exceed  an  operational  threshold 

This  is  the  probability  that  the  total  number  of  killed  VTs,  is  at  least  a  given 
percentage  of  their  initial  number. 

•  Expected  time  length  of  the  engagement 

This  is  the  expected  total  duration  of  the  engagement  -  until  all  UCAVs 
are  disposed. 

The  analysis  to  follow  will  focus  on  the  first  and  fourth  MOEs.  The 
first  MOE  measures  the  effectiveness  of  the  UCAVs;  the  last  one  measures  their 
time-efficiency,  which  provides  insight  about  the  possible  success  of  time  critical 
missions.  Specifically,  it  might  be  the  case  that  an  otherwise  optimal  UCAV 
mission  mix,  is  not  optimal  when  time  is  of  essence. 
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B.  DESCRIPTION  OF  THE  MODEL 


1.  Markov  Chain 

The  stochastic  model  for  the  combat  situation  described  above  is  a 
continuous  time  Markov  chain.  A  step  in  this  chain  is  defined  as  either  a 
detection  event  by  a  UCAV  or  a  UCAV  failure.  A  state  in  this  model  is  defined  by 
the  number  of  UCAVs  (of  each  type)  alive  and  the  number  of  targets  (of  each 
type)  still  alive.  A  transition  from  one  state  to  the  next  is  a  result  of  one  of  seven 
possible  events: 

•  A  type-A  UCAV  kills  a  type-1  target 

•  A  Type-A  UCAV  kills  a  type-2  target 

•  A  Type-B  UCAV  kills  a  type-1  target 

•  A  Type-B  UCAV  kills  a  type-2  target 

•  A  Type-A  UCAV  fails,  or  misses  a  VT,  or  is  wasted  on  a  NVT 

•  A  Type-B  UCAV  fails,  or  misses  a  VT,  or  is  wasted  on  a  NVT 

•  A  detection  is  recorded  but  none  of  the  six  events  mentioned  above 
occurs;  the  state  remains  unchanged 

2.  Assumptions 

We  make  the  following  assumptions: 

a.  Imperfect  Recognition 

The  sensor  of  each  type  of  UCAV  can  identify  the  status  of  the 
targets  (VT  or  NVT),  but  the  identification  is  not  perfect;  it  is  accurate  with  a  given 
probability,  which  depends  on  the  UCAV  type. 

b.  Communication  -  Coordination  and  Zero  Attack  Time 

There  is  communication  among  the  UCAVs  that  leads  to  perfect 
coordination,  in  the  sense  that  there  are  no  simultaneous  multiple  attacks  on  a 
single  target.  Also,  because  the  attack  time  is  short  compared  to  the  detection 
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time  and,  as  mentioned  above,  there  are  no  simultaneous  multiple  attacks  on  the 
same  target,  we  assume  that  the  attack  time  is  zero. 

c.  Fixed  Total  Number  of  UCA  Vs 

The  total  number  of  UCAVs  is  16.  This  way  we  have  a  convenient 
(and  rational)  upper  limit  on  the  model  complexity,  and  by  determining  the 
number  of  type-A  UCAVs  we  also  determine  the  number  of  type-B  ones. 

d.  Unlimited  Endurance 

UCAVs  are  assumed  to  have  enough  fuel  for  the  purposes  of  their 
mission,  and  therefore  they  can  never  crash  due  to  fuel  shortage. 

e.  Disposable  UCAVs 

As  mentioned  in  the  model  description,  the  UCAVs  are  disposable, 
and  they  never  run  out  of  fuel.  So,  even  if  all  the  VTs  are  destroyed,  the 
remaining  UCAVs,  if  any  left,  keep  loitering,  until  they  either  mistakenly  engage 
an  NVT  or  crash  due  to  mechanical  failure.  This  can  potentially  distort  the  values 
of  our  MOE  results  because  the  mission  duration  may  be  artificially  extended, 
after  all  the  targets  are  killed.  But,  this  could  only  be  a  problem  in  the  event 
where  there  is  a  significant  probability  that  all  the  VTs  are  destroyed.  As  we  see 
later  on,  this  is  a  very  rare  event  in  our  analysis.  To  make  this  point  clearer, 
consider  this  example:  there  are  only  two  targets,  and  presumably  six  UCAVs  are 
sufficient  for  destroying  them  all.  In  that  case,  if  we  were  to  employ  more  than 
six  UCAVs,  the  duration  of  the  mission  would  be  much  longer  since  the 
redundant  UCAVs  would  loiter  until  being  disposed,  and  the  Ev  would  be  the 
same  although  at  a  higher  UCAV  expense.  But  again,  in  the  DOEs  employed  in 
the  analysis,  this  situation  is  very  rare. 

On  the  other  hand,  note  that,  if  our  definition  of  absorbing  states 
changes  to  also  include  the  condition  ti  +  t2  =  0,  this  potential  distortion  of  MOE 
values  mentioned  above  would  never  be  the  case,  while  at  the  same  time,  we 
would  no  longer  have  to  assume  that  the  UCAVs  are  disposable.  Of  course,  for 
that  to  be  true,  the  target  recognition  has  to  be  perfect. 
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f.  Passive  Targets 

The  targets  do  not  shoot  back.  So,  there  is  no  UCAV  attrition  due  to 
enemy  fire.  This  assumption  implies  that  the  number  of  VTs  does  not  affect  the 
failure  rate  of  the  UCAVs.  Nonetheless,  we  can  always  incorporate  the  effects  of 
an  existing  air  defense  into  the  aggregated  UCAV  failure  parameters. 

g.  Exponential  Detection  Rate 

The  detections  follow  a  Poisson  process;  therefore  the  inter¬ 
detection  times  are  exponential  random  variables,  of  which  we  need  only  define 
the  mean.  This  is  a  reasonable  assumption,  because  we  can  think  of  targets  as 
forming  a  Poisson  field.  These  targets  move  randomly  in  the  target  area,  so  that 
they  are  always  spatially  distributed  according  to  a  spatial  Poisson  distribution. 

h.  Exponential  Failure  Rate 

Mechanical  failures  follow  a  Poisson  process  with  a  mean  that  can 
be  estimated  from  statistical  data  obtained  from  controlled  experiments.  In  the 
presence  of  enemy  air  defense,  this  failure  rate  could  also  incorporate  the  effects 
of  UCAV  attrition  by  the  air  defense. 

/.  Fixed  Probabilities 

All  the  probabilities  in  the  model  are  fixed  numbers  that  are 
independent  of  the  time  and  state  of  the  operation. 

j.  No  Partial  Damage 

Damage  is  not  accumulated  on  a  target.  If  a  target  has  not  been 
destroyed  during  an  attack,  it  is  considered  as  good  as  new. 

k.  Repeated,  Random,  Independent,  Memory-Less  Search 

UCAVs  search  independently,  in  a  random  search  pattern,  and  they 
have  no  memory  regarding  previous  detections.  Since  we  want  to  model  static 
and  moving  targets,  this  assumption  is  necessary,  to  avoid  implementing  a 
complex  tracking  algorithm. 
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3.  Notation 

a.  Detection  Exponential  Rates 

h\'.  the  detection  rate  of  type-i  UCAV,  i  =  A,  B. 

b.  Failure  Exponential  Rates 

0i :  the  failure  rate  of  type-i  UCAV,  i  =  A,  B. 

c.  Specificity 

n:  the  probability  that  a  type-i  UCAV  recognizes  a  NVT  as  such,  i  = 
A,  B. 

d.  Acquisition 

qy:  the  probability  that  a  type-i  UCAV  acquires  a  type-j  target,  given 
a  detection  of  such  a  target,  i  =  A,  B,  and  j  =  1 , 2. 

e.  Kill  Probabilities 

Pij :  the  probability  that  a  type-i  UCAV  kills  a  type-j  target,  given  an 
acquisition  of  such  a  target,  i  =  A,  B,  and  j  =  1 , 2. 

f.  Target  Values 

Tvi:  the  military  (operational)  value  of  a  type-i  target,  i  =  1 , 2. 

g.  Initial  Number  of  Targets 

Ti:  the  total  initial  number  of  type-i  targets,  i  =  1, 2,  3. 
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h.  Initial  Number  of  UCA  Vs 

Nil  the  initial  number  of  type-i  UCAVs,  i  =  A,  B. 

/.  State  Variables 

•  nj:  the  current  number  of  type-i  UCAVs,  i  =  A,  B. 

•  tj:  the  current  number  of  type-j  targets,  j  =  1 ,  2. 

4.  Mathematical  Formulation 
a.  Transitions 

Letting  the  initial  state  be  (NA,  NB,  T1;  T2),  and  the  current  state  (nA, 
nB,  t-i,  t2),  we  have  seen  in  section  1  above  that  there  are  seven  possibilities  for 
the  next  state  (feasibility  permitting). 

The  transition  probabilities  for  these  seven  cases,  along  with  a 
short  description  of  the  characteristics  of  the  transitions,  are  shown  below: 

•  Cases  1  -  4:  Type-i  UCAV  kills  Type-j  target,  i  =  A,  B,  j  =  1 , 2. 

‘iPaq. 

ixft+o,.)  ix 

k=l  m=l 


•  Cases  5  and  6:  Type-i  UCAV  fails,  or  misses  a  VT,  or  is  wasted  on  a  NVT, 
i  =  A,  B. 


„  /n  L)  \  S^k^ikC^'Pik)  3  2 

n'(0i+V)  »  -+(IVX<k)(l-r,)). 


£nt(kk+9l)  ^Tk 
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k=l 


k=l 
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•  Case  7:  State  remains  unchanged,  i  =  A,  B. 

,  ,, 

y _ ni^i _ k= i _ 1=1 _ i=i 

in 

k= 1  A:=l 

b.  Absorbing  States 

We  define  a  state  as  absorbing  when  nA  +  nB  =  0.  Note  that,  if  a 
state  with  ti  +  t2  =  0  is  also  specified  as  absorbing,  and  if  n  =  1  (i  =  A,  B),  we 
could  drop  the  disposable-UCAV  assumption. 

c.  Feasibility  Conditions 

The  feasibility  condition  for  the  states  is: 

NA-nA+NB-nB^Tl-tl+T2-t2  (or  equivalently  Ti+T2-Na-Nb  <  t|+t2-nA-nB )  and  all 
‘components’  of  each  state  must  be  non-negative.  Thus  we  can  discard  all 
states  that  do  not  satisfy  this  condition. 

d.  MOEs 

The  two  MOEs  to  be  calculated  in  the  analysis  are  shown  below: 

(1 )  Expected  value  of  killed  targets: 

Ev  =  E[#type-1  targets  killed]Tvi  +  E[#type-2  targets  killed]TV2 

(2)  Time  (expected  length  of  the  engagement) 

Note  that  both  MOEs  are  derived  from  the  Markov  chain  model; 
there  are  no  explicit  analytical  formulas  for  their  calculation. 
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C.  IMPLEMENTATION  OF  THE  MODEL 

1.  General 

The  continuous  time  Markov  chain  model  and  all  the  subsequent  models 
associated  with  the  analysis  are  implemented  using  the  Matlab  programming 
environment. 

2.  Matlab  Code  Objectives 

The  code  written,  accomplishes  many  tasks: 

•  Getting  input  and  assigning  values  to  the  parameters 

•  Finding  and  counting  the  feasible  states 

•  Distinguishing  between  transient  and  absorbing  states 

•  Mapping  and  populating  the  transition  probability  matrix  P 

•  Deriving  results  from  Markov  chain  theory,  with  respect  to  P 

•  Calculating  the  MOE  values 

•  Checking  for  errors  during  run-time 

•  Generating  output 

3.  Code 

The  Matlab  code  written  for  this  model  implementation  appears  in  the 
Appendix. 

D.  BROAD  EXPERIMENTAL  SET-UP  AND  DATA  ANALYSIS 

We  apply  a  DOE  scheme  for  building  a  regression  meta-model,  and 
maximizing  the  information  obtained  from  a  given  number  of  runs  of  the  model. 
The  DOE  of  choice  is  NOLH  for  reasons  explained  in  Chapter  II.  The  basic 
concepts  and  terminology  used  herein  are  also  presented  in  Chapter  II. 
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1 .  Factors 

a.  Decision  Factors 

We  are  primarily  interested  in  exploring  optimal  mixes  of  the  two 
types  of  UCAVs.  Since  we  assume  a  total  number  of  16  UCAVs,  NB  =  16  -  NA, 
and  therefore  we  only  have  one  decision  factor,  NA. 


b.  Environmental  and  Design  Factors,  and  Their  Ranges 

(1 )  Environmental  (noise)  factors 


•  T-i  e  [3,  7] 

•  T2  e  [3,  7] 

•  T3  e  [0,  28] 

•  Tvi  =  1 

•  Tv2  e  [0,  1] 


[discrete] 

[discrete] 

[discrete] 

[fixed] 

[continuous] 


The  absolute  expected  total  value  of  the  engagement  is  not 
important  and  therefore  it  can  be  scaled  such  that  Tvi  is  fixed  at  the  value  1,  and 
only  Tv2  varies.  By  doing  that,  we  reduce  the  dimensionality  of  the  model  by  one, 
thus  making  the  DOE  and  the  subsequent  analysis  less  cluttered. 


(2) 


qA1  e  [0.5,  1] 
given  a  detection 


qA2 


,C1b2 


1.2 

liven  a  detection 

Iai 


3bi 


,c1ai 


1.2 

^iven  a  detection 
1b2  e  [0.5,  1] 
}iven  a  detection 


UCAV-design  (noise)  factors 

(probability  that  a  type-A  UCAV  acquires  a  type-1  target, 
of  such  a  target) 

(probability  that  a  type-A  UCAV  acquires  a  type-2  target, 
of  such  a  target) 

(probability  that  a  type-B  UCAV  acquires  a  type-1  target, 
of  such  a  target) 

(probability  that  a  type-B  UCAV  acquires  a  type-2  target, 
of  such  a  target) 
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•  rA  e  [0.5,  1.0]  (probability  that  a  type-A  UCAV  recognizes  a  NVT  as 

such) 

•  rB  e  [0.5,  1.0]  (probability  that  a  type-B  UCAV  recognizes  a  NVT  as 

such) 

•  Pai  e  [0.4,  1 .0]  (probability  that  a  type-A  UCAV  kills  a  type-1  target,  given 
an  acquisition  of  such  a  target) 

•  pA2  e  Pbi  p  (probability  that  a  type-A  UCAV  kills  a  type-2  target, 

L 1-2  B“J 

given  an  acquisition  of  such  a  target) 

•  pB1  G  PalPii  (probability  that  a  type-B  UCAV  kills  a  type-1  target, 

L 1-2  A1J 

given  an  acquisition  of  such  a  target) 

•  pB2  e  [0.4,  1 .0]  (probability  that  a  type-B  UCAV  kills  a  type-2  target,  given 
an  acquisition  of  such  a  target) 

•  Aa  e  [0.2,  1]  (detection  rate  of  type-A  UCAV) 

•  AB  e  j^,AA  (detection  rate  of  type-B  UCAV) 

•  0A  e  [0.005,  0.03]  (failure  rate  of  type-A  UCAV) 

•  0B  e  ^a_  0  (failure  rate  of  type-B  UCAV) 

1.2 

Note  that  some  parameters  (e.g.,  qAi  and  qBi)  are  correlated.  The 
reason  for  doing  that  is  to  explore  whether  reducing  the  dimensionality  of  the 
DOE  still  conveys  similar  analysis  results  or  not.  If  so,  fewer  varying  parameters 
will  be  examined  in  any  consequent  analysis  thereon,  without  any  loss  of 
generality. 

Also  note  that  restricting,  for  example  qB1,  to  be  1  to  1.2  times  less 
than  qAi,  is  equivalent  to  determining  that  qBi  is  83%  to  100%  of  qAi.  The  same 
is  true  for  all  the  other  pairs  of  correlated  parameters.  We  prefer  showing  the  1 .2 
ratio  factor  instead  of  a  percentage  because  in  the  next  section  we  use  this  factor 
for  narrowing  down  the  DOE  (we  call  it  a  handicap). 
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Absent  hard  data,  the  ranges  for  the  factor  values  could  only  be 
educated  guesses,  based  on  the  existing  literature.  However,  we  intentionally 
made  these  ranges  quite  broad,  because  the  NOLH  DOE  (see  Ch.  II)  that  we 
employ  for  the  analysis  gives  us  flexibility,  which  facilitates  exploring  a  broader 
range  of  factor  values.  In  Section  F  we  set  up  a  narrowed  down  experimental 
design  that  significantly  reduces  the  dimensionality  of  our  model,  while  at  the 
same  time,  maintains  the  potential  for  robust  inference,  and  clear  conclusions. 

We  tried  not  to  give  any  UCAV  a  clear  overall  advantage  over  the 
other  one  (otherwise  the  optimal  UCAV  mix  is  trivial).  So,  a  type-A  UCAV  is 
more  effective  against  a  type-1  target,  and  a  type-B  UCAV  is  more  effective 
against  a  type-2  target.  Also,  a  type-A  UCAV  has  a  higher  detection  rate,  but  is 
more  failure-prone  than  a  type-B  UCAV.  The  targets,  too,  have  some 
differences.  Type-1  targets  are  more  valuable  but  harder  to  acquire  than  type-2 
targets. 

We  seek  to  identify  situations  where  the  mission  mix  affects  the 
outcome  of  the  engagement,  despite  this  balanced  setup. 

2.  Design  of  Experiments 

We  use  a  129-NOLH  design  for  the  18  (varying)  noise  factors  and  we 
cross  it  with  the  17  discrete  levels  of  the  (unique)  decision  factor  NA.  This  gives 
129  x  17  =  2,193  design  points  in  total. 

3.  Batch-Running  the  Model 

Since  MATLAB®  can’t  run  in  command  line  mode,  we  cannot  use  a  batch 
file  approach  to  automate  the  2,193  runs.  Instead,  we  feed  an  Excel®  file  that 
contains  all  the  design  points  into  the  MATLAB  workspace,  and  the  MATLAB 
code  sequentially  reads  the  parameter  values  corresponding  to  each  design 
point,  runs  the  continuous  time  Markov  model,  and  generates  the  output  values 
which  are  then  saved  to  another  Excel  file.  All  this  input  and  output  (I/O)  is 
performed  automatically  by  the  MATLAB  code  without  the  need  for  any  additional 
setup. 
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4.  Analysis  and  Results 

a.  Course  of  Analysis 

The  Markov  chain  model  employs  various  design,  operational,  and 
decision  factors.  The  relations  between  these  parameters  and  the  MOEs  are 
mapped,  in  order  to  gain  insights  regarding  the  design  of  the  UCAVs  and  their 
tactical  employment.  The  statistical  package  JMP®  is  used  to  evaluate  the  MOE 
outputs  from  the  model  runs  based  on  sets  of  inputs  that  are  determined 
according  to  the  NOLH  DOE  principles.  Since  we  are  interested  in  the  optimal 
UCAV  mix,  we  plot  the  MOE  values  as  functions  of  NA,  in  order  to  observe  how 
significant  the  decision  factor  is. 

Also  we  employ  regression  analysis  as  our  main  tool  for  meta¬ 
modeling  the  mathematical  model.  We  construct  various  regression  models  of 
the  MOEs,  which  include  linear,  quadratic,  and  second  degree  interaction  terms, 
for  investigating  whether  a  simpler  meta-model  is  more  efficient  than  a  more 
accurate  one,  by  being  able  to  capture  most  of  the  modeling  situation  with  only  a 
subset  of  the  factors.  If  the  regression  accounts  for  most  of  the  variability  of  our 
model,  it  can  be  assigned  the  role  of  a  ‘hard  and  fast’  substitute  for  the  model. 
This  is  convenient  when  the  model  is  not  available,  or  when  the  time  available  for 
running  the  model  is  limited.  Also,  the  regression  function  transparently  shows 
which  factors  have  an  effect  on  the  MOE,  how  big  the  effect  is,  and  in  what 
direction. 


b.  Analysis 

First  we  explore  how  does  mission  mix  (i.e. ,  the  NA  decision  factor) 
affect  the  average  value  of  the  Ev  MOE.  As  we  see  in  Figure  1,  although  the  Ev 
tends  to  be  higher  for  larger  NA  values,  the  effect  is  insignificant. 

Throughout  we  follow  the  convention  that  plots  be  accompanied  by 

their  respective  data  tables,  which  in  some  cases  contain  also  some  additional 

information.  Here,  in  Table  1,  the  Ev  values  are  shown  along  with  their  standard 

deviation  values  which  are  fairly  constant,  ratifying  the  use  of  regression 

analysis.  Also,  we  can  see  the  extreme  values  and  the  range  of  Ev  values  for 

each  Na.  They  all  moderately  increase  with  NA,  showing  that  if  NA  has  any  effect 
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on  Ev  it  has  to  be  positive,  even  though  the  overall  effect  appears  insignificant  in 
this  case.  This  insignificance  is  attributed  to  the  design  of  the  1 8-noise-facrtor 
fairly  balanced  combat  situation. 

Note  that  the  plot  in  Figure  1  is  drawn  on  a  magnified  scale,  and  the 
nature  of  the  relation  (e.g.,  quadratic)  is  revealed,  due  to  the  very  limited  range  of 
Ev  values;  this  is  indifferent  to  our  conclusions  though.  In  this  case,  actually,  our 
interpretation  is  that  the  mission  mix  makes  no  practical  difference. 


Na 

Mean(Ev) 

StDev(Ev) 

Min(Ev) 

Max(Ev) 

Range(Ev) 

0 

4.112 

1.670 

1.177 

8.958 

7.781 

1 

4.131 

1.643 

1.243 

8.773 

7.530 

2 

4.147 

1.618 

1.308 

8.587 

7.279 

3 

4.163 

1.597 

1.362 

8.399 

7.037 

4 

4.178 

1.580 

1.362 

8.361 

6.999 

5 

4.191 

1.565 

1.362 

8.470 

7.108 

6 

4.203 

1.555 

1.362 

8.576 

7.214 

7 

4.214 

1.548 

1.362 

8.681 

7.319 

8 

4.223 

1.544 

1.362 

8.784 

7.422 

9 

4.232 

1.545 

1.362 

9.061 

7.699 

10 

4.239 

1.549 

1.362 

9.420 

8.058 

11 

4.245 

1.556 

1.362 

9.764 

8.402 

12 

4.250 

1.567 

1.362 

10.088 

8.727 

13 

4.254 

1.581 

1.361 

10.389 

9.028 

14 

4.256 

1.598 

1.361 

10.662 

9.300 

15 

4.257 

1.617 

1.361 

10.904 

9.542 

16 

4.258 

1.638 

1.361 

11.113 

9.752 

Table  1 .  Statistics  on  the  Ev  MOE  for  the  different  NA  values 


Figure  1 .  The  average  Ev  MOE  values  as  a  function  of  NA 
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Next,  we  explore  how  NA  affects  the  mean  time  of  the  operation 
(second  MOE).  As  we  can  see  in  Figure  2,  the  expected  time  tends  to  be  shorter 
for  larger  NA  values.  So,  as  with  Ev,  this  MOE  too,  gets  a  more  favorable  value 
as  Na  increases.  Note  though,  that  the  effect  of  NA  on  Time  is  more  significant 
than  it  is  on  Ev. 

In  Table  2,  the  Time  values  are  shown  along  with  the  respective 
standard  deviation  values,  which  are  not  constant,  making  the  use  of  regression 
analysis  for  this  MOE  less  appropriate.  Also,  we  can  see  the  extreme  values  and 
the  range  of  Time  values  for  each  NA.  When  NA  is  about  its  middle  value,  the 
Time  values  are  more  consistent.  Because  of  this  behavior,  we  conclude  that 
although  the  Time  is  shorter  for  larger  NA,  it  is  wiser  to  choose  an  intermediate 
Na  value  (like  9,  in  this  case),  since  this  gives  a  worst  case  (max)  Time  value 
which  is  almost  half  the  value  we  would  get  if  NA  was  closer  to  its  extreme  points 
(0  and  16). 


NA 

Mean(Time) 

StDev(Time) 

Min(Time) 

Max(Time) 

Range(Time) 

0 

22.668 

15.434 

6.623 

117.669 

111.045 

1 

22.581 

14.727 

6.815 

111.031 

104.216 

2 

22.490 

14.073 

7.002 

104.418 

97.416 

3 

22.397 

13.482 

7.184 

97.863 

90.679 

4 

22.301 

12.962 

7.362 

91.403 

84.040 

5 

22.203 

12.524 

7.536 

85.067 

77.530 

6 

22.104 

12.178 

7.590 

78.883 

71.293 

7 

22.005 

11.934 

7.558 

72.875 

65.317 

8 

21.906 

11.800 

7.526 

67.057 

59.531 

9 

21.808 

11.786 

7.493 

61.439 

53.946 

10 

21.711 

11.904 

7.460 

62.108 

54.648 

11 

21.617 

12.167 

7.427 

65.629 

58.202 

12 

21.527 

12.592 

7.394 

69.161 

61.768 

13 

21.439 

13.196 

7.289 

79.896 

72.607 

14 

21.353 

13.997 

7.124 

92.774 

85.650 

15 

21.265 

15.003 

6.956 

107.535 

100.579 

16 

21.170 

16.210 

6.578 

123.934 

117.356 

Table  2.  Statistics  on  the  Time  MOE  for  the  different  NA  values 
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Figure  2.  The  average  Time  MOE  values  as  a  function  of  Na 


Next  we  build  three  regression  meta-models  of  the  mathematical 
model.  They  describe  the  Ev  MOE  only  (since  due  to  heteroscedasticity  we 
deemed  Time  inappropriate  for  regression).  The  first  regression  only  employs 
the  (linear)  model  factors,  the  second  one  additionally  employs  quadratic  terms 
(i.e. ,  the  model  factors  squared),  and  the  third  one  additionally  employs  all 
possible  second  degree  interactions  of  the  model  factors.  It  is  obvious  that  the 
third  meta-model  has  the  most  descriptive  power,  but  we  explore  the  other  two 
cases  because  simplicity  and  fewer  terms  are  desirable  regression  attributes 
(they  aid  the  interpretation)  even  when  the  descriptive  power  falls  a  little  bit 
behind. 

(1 )  Regression  with  linear  terms  only 

Here,  the  18  noise  factors  and  the  single  decision  factor  are 
initially  added  to  the  regression  which  sequentially  eliminates  the  factors  that  are 
not  significant.  The  R2  (as  well  as  the  adjusted  R2)  is  about  86%,  which  is  fairly 
high.  In  Table  3  we  see  the  values  of  the  parameter  estimates  for  this 
regression.  Notice  how  small  the  standard  error  for  each  of  the  13  estimates  is, 
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making  the  regression  more  robust.  Also,  note  that  Na  is  eliminated  as  a  factor 
(i.e.,  it  is  statistically  insignificant),  as  we  might  have  expected  from  the  previous 
analysis. 


Response  Ev 
Summary  of  Fit 


RSquare 

0.858 

RSquare  Adj 

0.857 

Root  Mean  Square  Error 

0.598 

Mean  of  Response 

4.209 

Observations  (or  Sum  Wgts) 

2193 

Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept  -6.911 

0.171 

-40.44 

<.0001 

Ti 

0.385 

0.009 

42.55 

<.0001 

t2 

0.104 

0.009 

11.59 

<.0001 

t3 

-0.075 

0.002 

-47.78 

0.0000 

T  V2 

2.910 

0.044 

66.27 

0.0000 

qAi 

1.245 

0.088 

14.17 

<.0001 

9b2 

0.458 

0.088 

5.22 

<.0001 

Ta 

2.545 

0.088 

28.98 

<.0001 

Tb 

2.631 

0.088 

29.87 

<.0001 

Pai 

1.325 

0.341 

3.88 

0.0001 

Pbi 

1.618 

0.366 

4.43 

<.0001 

PA2 

1.748 

0.078 

22.30 

<.0001 

Aa 

0.532 

0.055 

9.70 

<.0001 

0a 

-17.883 

1.909 

-9.37 

<.0001 

Table  3.  Regression  with  linear  terms  only  output 


(2)  Regression  with  quadratic  terms  also 


This  time,  the  squared  factors  are  also  considered  for  the 
regression.  After  the  elimination  process,  we  are  left  with  35  estimates.  The 


corresponding  terms  are: 


T  T2  T  T2T  T2  T 
xl’xl  ’x2’x2  x3’x3  5  1 ' 


rp  2  2  2  2  2 

V2  ’  *  V2  Am  Aa\  ’^Bl  Ab\  ’^1b2  Aa2  Aa2  ^ A  ’**A  ’ 


1  >Pb2  ?Pb2  >Pa2  ?Pa2  $A  ^ A  $ 


A2 

B?UB* 


The  parameter  estimates  for  these  terms  are  shown  in  Table  4.  The  R2  value  is 
90%  in  this  case.  That  value  is  not  much  larger  than  86%,  and  therefore  the  use 
of  22  additional  terms  does  not  seem  to  be  justified. 

Note  how  the  quadratic  terms  are  centered  about  the  mean  value  of  their 
corresponding  linear  term.  For  example,  the  average  Ti  value  is  5,  and  therefore 
the  quadratic  Ti  term  is  (Ti  -  5)2  instead  of  Ti2.  This,  according  to  regression 
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theory,  leads  to  a  less  biased  regression  equation.  Besides  that,  note  the 
elimination  of  Na  from  this  regression  too. 

(3)  Regression  with  all  linear,  quadratic,  and  second- 
degree  interaction  terms 

This  regression  considers  all  the  factors,  their  squares,  and 
their  possible  second  degree  interactions.  After  the  elimination  process  the 
terms  left  in  the  regression  are  126  which  render  the  use  of  this  regression 
unreasonable,  even  though  the  R2  has  climbed  up  to  98%. 

c.  Results 

By  employing  this  broad  experimental  design  setup,  it  is  obvious 
that,  due  to  the  balancing  of  the  parameter  values,  there  appears  to  be  no  effect 
of  Na  on  Ev.  And  although  there  is  some  effect  on  the  other  MOE,  Time,  it  does 
not  seem  to  be  that  important  either.  Nevertheless,  for  Time,  we  reached  the 
important  conclusion  that  absent  hard  data  it  is  better  to  employ  a  balanced  mix 
of  UCAVs,  instead  of  a  biased  one  where  most  of  the  UCAVs  are  of  one  type. 

In  the  next  section,  we  try  to  decrease  the  overall  noise,  by 
employing  a  different  experimental  setup.  This  is  based  on  the  current  one,  but  it 
is  narrower,  by  using  the  ‘handicap’  concept,  and  selecting  discrete  values  for  the 
T v2  factor. 

E.  NARROWER  EXPERIMENTAL  SET-UP  AND  DATA  ANALYSIS 

In  this  DOE,  the  varying  factors  are  decreased  down  to  13  as  opposed  to 
19  factors  in  the  previously  discussed  DOE.  The  other  six  factors  are  correlated 
to  six  of  the  13  independent  factors,  according  to  a  handicap  that  is  decided  to  be 
equal  to  1.2.  A  smaller  handicap  wouldn’t  reveal  the  various  factor  effects  as 
articulately,  whereas  a  larger  one  would  just  provide  for  the  same  insight. 
Therefore,  we  can  employ  a  1.2  handicap  without  loss  of  generality  on  our 
analysis  conclusions. 
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Response  Ev 
Summary  of  Fit 


RSquare 

0.900 

RSquare  Adj 

0.898 

Root  Mean  Square  Error 

0.504 

Mean  of  Response 

4.209 

Observations  (or  Sum  Wgts) 

2193 

Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

-7.324 

0.155 

-47.16 

0.0000 

Ti 

0.384 

0.008 

50.29 

0.0000 

(Ti-5)2 

0.032 

0.008 

4.12 

<.0001 

t2 

0.106 

0.008 

13.94 

<.0001 

(T2-5)2 

-0.034 

0.009 

-3.89 

0.0001 

t3 

-0.075 

0.001 

-56.40 

0.0000 

(T3-14.015)2 

0.0006 

0.0002 

2.77 

0.0056 

T  V2 

2.951 

0.037 

79.40 

0.0000 

(Tv2-0.500)2 

2.398 

0.182 

13.17 

<.0001 

9ai  , 

0.392 

0.296 

1.33 

0.1847 

(qn-0.750)2 

8.661 

1.177 

7.36 

<.0001 

qBi 

1.045 

0.316 

3.30 

0.0010 

(qei-0.684)2 

-4.249 

1.254 

-3.39 

0.0007 

qB2 

0.775 

0.290 

2.68 

0.0075 

0A2 

-0.415 

0.306 

-1.36 

0.1753 

(qA2-0.684)2 

1.788 

0.730 

2.45 

0.0144 

rA 

2.531 

0.074 

34.11 

<.0001 

(rA-0.750)2 

2.541 

0.744 

3.41 

0.0007 

Tb 

2.600 

0.075 

34.74 

<.0001 

(re-0.750)2 

3.739 

0.702 

5.33 

<.0001 

pAi 

-0.075 

0.331 

-0.23 

0.8207 

(pAi-0.700)2 

6.727 

1.168 

5.76 

<.0001 

Pbi 

3.210 

0.358 

8.97 

<.0001 

(Pbi-0.638)2 

-15.913 

1.366 

-11.65 

<.0001 

PB2 

1.078 

0.326 

3.30 

0.0010 

(PB2-0.700)2 

-2.867 

1.059 

-2.71 

0.0069 

PA2 

0.626 

0.353 

1.77 

0.0761 

(Pa2-0.638)2 

5.258 

1.146 

4.59 

<.0001 

Aa 

-0.192 

0.415 

-0.46 

0.6441 

(Aa-0.600)2 

2.168 

0.955 

2.27 

0.0234 

Ab 

0.800 

0.455 

1.76 

0.0790 

(Ab-0.547)2 

-5.114 

1.086 

-4.71 

<.0001 

eA 

-60.146 

14.240 

-4.22 

<.0001 

(9a-0.017)2 

5483.747 

1331.215 

4.12 

<.0001 

0B 

47.936 

15.561 

3.08 

0.0021 

(Ob-0.016)2 

-6208.849 

1542.055 

-4.03 

<.0001 

Table  4.  Regression  with  linear  and  quadratic  terms  output 
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1 .  Factors 

a.  Decision  Factor 

Again,  we  only  have  one  decision  factor,  Na,  which  determines  the 

UCAV  mix. 

b.  Environmental  and  Design  Factors,  and  Their  Ranges 

(1 )  Environmental  (noise)  factors 


•  Tie  [3, 7]  [discrete] 

•  T2  e  [3, 7]  [discrete] 

•  T3  e  [0, 28]  [discrete] 

•  Tvi  =  1  [fixed] 

•  TV2  e  {0,  0.5,  1}  [discrete] 


Note  that  when  TV2  (which  is  now  discrete)  is  equal  to  0,  then  the 
Tvi/TV2  ratio  goes  to  infinity.  Killing  type-2  targets  adds  no  value  to  the  military 
operation.  On  the  other  hand,  when  TV2  is  equal  to  1,  that  ratio  becomes  1, 
meaning  that  both  types  of  targets  have  the  same  military  value  and  thus  their 
value  is  not  a  factor  mission-wise. 

(2)  UCAV-design  (noise)  factors 

This  experimental  setup  has  a  reduced  dimensionality 
compared  to  the  previous  broad  setup.  For  accomplishing  this,  the  correlated 
parameters  (e.g.,  qAi  and  qBi)  have  a  fixed  relation,  called  a  handicap,  instead  of 
a  randomly  varying  relation.  So,  the  handicap  is  a  fixed  coefficient  by  which 
some  parameters  are  inferior  compared  to  their  respective  correlated 
parameters.  In  the  subsequent  analysis,  we  also  explore  the  effect  of  different 
handicap  values  (1.0,  1.1,  1.2)  on  the  inference  significance.  The  default 
handicap  is  1 .2  (corresponds  to  83%  inferiority). 

•  qAi  e  [0.5,  1]  [continuous] 

•  qA2  =  Yy  [correlated  with  qB2] 
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n  —  ^A1 

qB1  -  — 

[correlated  with  qAi] 

qB2e  [0.5,  1] 

[continuous] 

rAe  [0.5,  1.0] 

[continuous] 

rB  e  [0.5,  1 .0] 

[continuous] 

pAi  e  [0.4,  1.0] 

[continuous] 

rv  -  Pb2 

Pa2  "IT 

[correlated  with  pB2] 

n  -  PAi 

PB1  "II 

[correlated  with  pAi] 

pB2  e  [0.4,  1.0] 

[continuous] 

Aa  e  [0.2,  1] 

[continuous] 

Ab  =  — 

1.2 

[correlated  with  Aa] 

•  0A  e  [0.005,  0.03]  [continuous] 

•  Os  =  ^  [correlated  with  0A] 

This  narrower  experimental  design  above,  has  a  significantly 
reduced  dimensionality  compared  to  the  previous  broad  DOE,  while  at  the  same 
time,  as  we  will  see,  conveys  the  same  types  of  results  for  Ev,  when  analysis  is 
done.  For  Time,  though,  we  get  different  results,  which  now  are  more  consistent 
with  the  results  for  Ev. 

Again,  the  properties  of  the  UCAVs  and  the  targets,  are  balanced, 
enabling  the  exploration  of  non-trivial  UCAV  mixes. 

2.  Design  of  Experiments 

We  use  a  33-NOLH  design  for  the  1 1  (varying)  noise  factors  and  we  cross 
it  with  the  3  discrete  levels  of  the  TV2  noise  factor  as  well  as  the  17  discrete  levels 
of  the  decision  factor  NA.  This  gives  33  x  3  x  17  =  1,683  design  points  in  total, 
30%  less  than  in  the  initial  129  x  17  DOE,  while  at  the  same  time  we  have  very 
enhanced  resolution  (i.e.,  transparency  for  the  effects  of  NA  and  the  Tvi/TV2  ratio) 
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due  to  the  double  crossing  in  the  design.  If  the  design  was  not  crossed,  then  a 
possible  effect  on  some  MOE  would  not  be  clearly  attributed  to  either  the 
decision  factor  or  a  noise  factor  or  the  Tvi/TV2  ratio,  without  additional 
investigation.  This  kind  of  design  leads  to  clearer  cause-and-effect  conclusions, 
at  the  expense  of  more  design  points  than  an  equivalent  single  DOE  would  have. 

3.  Analysis  and  Results 

a.  Course  of  Analysis 

As  in  the  previous  section,  we  are  interested  in  the  optimal  UCAV 
mix,  and  therefore  we  plot  the  MOE  values  as  functions  of  Na,  in  order  to  observe 
how  significant  the  decision  factor  is.  We  investigate  the  role  of  Tv2  to  the 
significance  of  the  NA  effect.  It  turns  out  that  TV2  affects  only  Ev,  not  Time.  The 
results  we  get  are  consistent  with  the  previous  section  analysis  results,  only  more 
pronounced.  So,  the  use  of  the  computationally  less  expensive  narrowed  down 
DOE  is  justified. 

Also  we  employ  regression  analysis  as  our  main  tool  for  meta¬ 
modeling  the  mathematical  model.  We  construct  various  regression  models  of 
the  MOEs.  Again,  arguably,  the  regression  function  of  choice  is  the  regression 
with  linear  terms  only. 

b.  Analysis 

Initially,  we  explore  how  the  mission  mix  (i.e. ,  the  NA  decision 
factor)  affects  the  value  of  the  Ev  MOE,  for  various  handicap  values  and  Tvi/Tv2 
ratios.  What  we  first  see  is  that  the  higher  the  TVi/TV2  ratio  the  better  the  effect  of 
Na  on  Ev  shows.  Given  a  high  Tvi/Tv2  ratio,  larger  handicaps  give  more 
noticeable  effects.  Therefore  an  initial  conclusion  here  is  that  we  should  choose 
a  large  enough  handicap  for  the  effects  to  show,  and  that  when  the  operational 
values  of  the  two  types  of  targets  are  close,  the  effects  tend  to  be  masked  no 
matter  what  the  handicap  is.  In  the  subsequent  analysis,  whenever  a  handicap 
has  to  be  employed  it  will  take  on  a  value  of  1 .2.  Note  that  the  selection  of  a  1 .2 
handicap  value  is  arbitrary  (for  example  it  could  be  larger)  but  without  introducing 
any  loss  of  generality.  The  above  conclusions  are  backed  up  by  the  plots  in 
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Figures  3  to  5.  As  always,  the  plots  are  accompanied  by  their  respective  data 
tables  (Table  5  to  7).  Note  that  Figure  3  has  TV2  =  0,  Figure  4  has  TV2  =  0.5,  and 
Figure  5  has  TV2  =  1 .  Tvi  is  fixed  to  1  as  per  the  design  explained  in  Section  1b. 

The  conclusion  here  is  that  a  larger  Na  is  always  better  for  the 
military  value  of  the  operation,  but  we  should  only  strive  for  it  when  the 
differences  between  the  two  UCAV  types  are  substantial  and  the  operational 
values  of  the  two  types  of  targets  are  quite  distant. 


handicap  =  1.1 

Tv2=0 

Na 

Ev 

0 

2.637 

1 

2.660 

2 

2.683 

3 

2.705 

4 

2.725 

5 

2.745 

6 

2.764 

7 

2.782 

8 

2.800 

9 

2.816 

10 

2.832 

11 

2.847 

12 

2.861 

13 

2.874 

14 

2.887 

15 

2.899 

16 

2.911 

handicap  =  1.2 

Tv2=0 

Na 

Ev 

0 

2.395 

1 

2.438 

2 

2.478 

3 

2.518 

4 

2.556 

5 

2.593 

6 

2.629 

7 

2.664 

8 

2.697 

9 

2.730 

10 

2.761 

11 

2.791 

12 

2.821 

13 

2.849 

14 

2.877 

15 

2.904 

16 

2.931 

handicap  =  1.0 

Tv2=0 

Na 

Ev 

0 

2.910 

1 

2.914 

2 

2.918 

3 

2.921 

4 

2.924 

5 

2.926 

6 

2.927 

7 

2.927 

8 

2.927 

9 

2.925 

10 

2.923 

11 

2.920 

12 

2.916 

13 

2.912 

14 

2.906 

15 

2.900 

16 

2.893 

Table  5.  Ev  as  a  function  of  Na  when  Tv2  =  0  for  three  different  handicaps. 
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■  handicap^!. 2  ■  handicap^!.  1  ^^^^^^"handicap=1.0 


Figure  3.  Ev  as  a  function  of  Na  when  Tv2  =  0  for  three  different  handicaps. 


handicap  =  1.0 

Tv2=0.5 

Na 

Ev 

0 

4.364 

1 

4.371 

2 

4.377 

3 

4.381 

4 

4.385 

5 

4.388 

6 

4.389 

7 

4.390 

8 

4.389 

9 

4.388 

10 

4.385 

11 

4.381 

12 

4.376 

13 

4.369 

14 

4.362 

15 

4.354 

16 

4.344 

handicap  =  1.2 

Tv2=0.5 

Na 

Ev 

0 

3.869 

1 

3.897 

2 

3.924 

3 

3.949 

4 

3.972 

5 

3.993 

6 

4.013 

7 

4.031 

8 

4.047 

9 

4.061 

10 

4.074 

11 

4.085 

12 

4.095 

13 

4.103 

14 

4.109 

15 

4.114 

16 

4.118 

handicap  =  1.1 

Tv2=0.5 

Na 

Ev 

0 

4.101 

1 

4.117 

2 

4.132 

3 

4.147 

4 

4.160 

5 

4.171 

6 

4.182 

7 

4.191 

8 

4.199 

9 

4.206 

10 

4.212 

11 

4.216 

12 

4.219 

13 

4.222 

14 

4.223 

15 

4.223 

16 

4.222 

Table  6.  Ev  as  a  function  of  NA  when  TV2  =  0.5  for  three  different  handicaps. 
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■  “  ■  handicap=1.2  "  handicap=1 .1  handicap=1 .0 


Figure  4.  Ev  as  a  function  of  NA  when  TV2  =  0.5  for  three  different  handicaps. 


handicap  =  1.0 

TV2=1 

Na 

Ev 

0 

5.819 

1 

5.828 

2 

5.835 

3 

5.841 

4 

5.846 

5 

5.850 

6 

5.852 

7 

5.853 

8 

5.852 

9 

5.850 

10 

5.846 

11 

5.841 

12 

5.835 

13 

5.827 

14 

5.818 

15 

5.807 

16 

5.795 

handicap  =  1.2 

TV2=1 

Na 

Ev 

0 

5.343 

1 

5.357 

2 

5.369 

3 

5.379 

4 

5.387 

5 

5.393 

6 

5.396 

7 

5.397 

8 

5.396 

9 

5.393 

10 

5.387 

11 

5.379 

12 

5.369 

13 

5.356 

14 

5.341 

15 

5.324 

16 

5.305 

handicap  =  1.1 

TV2=1 

Na 

Ev 

0 

5.564 

1 

5.574 

2 

5.582 

3 

5.589 

4 

5.594 

5 

5.597 

6 

5.599 

7 

5.600 

8 

5.599 

9 

5.596 

10 

5.592 

11 

5.586 

12 

5.578 

13 

5.569 

14 

5.558 

15 

5.546 

16 

5.533 

Table  7.  Ev  as  a  function  of  NA  when  TV2  =  1  for  three  different  handicaps. 
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6.0 

5.8 

avg  Ev 

5.6 

5.4 

5.2 


“  “  ■  handicap=1.2  —  —  handicap=1.1  handicap=1.0 


Figure  5.  Ev  as  a  function  of  Na  when  Tv2  =  1  for  three  different  handicaps. 


Adopting  a  handicap  of  1 .2,  this  time  we  estimate  the  Ev  values 
along  with  other  statistical  quantities  for  different  TV2  values  as  a  function  of  NA. 
This  data  is  shown  in  Tables  9  to  11.  By  examining  the  standard  deviation 
columns,  we  observe  homoscedasticity  and  we  conclude  that  a  regression  would 
be  appropriate  on  Ev  as  a  function  of  all  the  model  parameters.  This  is  done  and 
explained  later  on  in  this  Section. 


Na 

Mean(Ev) 

StDev(Ev) 

Min(Ev) 

Max(Ev) 

Range(Ev) 

0 

2.395 

1.100 

0.614 

5.614 

5.000 

1 

2.438 

1.091 

0.646 

5.589 

4.943 

2 

2.478 

1.084 

0.678 

5.565 

4.887 

3 

2.518 

1.078 

0.710 

5.541 

4.831 

4 

2.556 

1.074 

0.741 

5.516 

4.775 

5 

2.593 

1.072 

0.771 

5.491 

4.720 

6 

2.629 

1.071 

0.802 

5.467 

4.665 

7 

2.664 

1.071 

0.832 

5.443 

4.611 

8 

2.697 

1.072 

0.861 

5.419 

4.558 

9 

2.730 

1.075 

0.890 

5.395 

4.505 

10 

2.761 

1.078 

0.919 

5.372 

4.453 

11 

2.791 

1.083 

0.947 

5.349 

4.402 

12 

2.821 

1.089 

0.975 

5.326 

4.351 

13 

2.849 

1.096 

1.003 

5.305 

4.302 

14 

2.877 

1.104 

1.030 

5.309 

4.280 

15 

2.904 

1.113 

1.027 

5.413 

4.386 

16 

2.931 

1.122 

1.013 

5.513 

4.500 

Table  8.  Statistics  on  the  Ev  MOE  for  the  different  NA  values  for  TV2  =  0. 
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Next,  we  explore  how  Na  affects  the  mean  time  of  the  operation 
(Time  MOE).  The  trend  is  the  same  for  different  handicaps,  therefore  we  display 
only  the  findings  for  a  handicap  of  1 .2.  Of  course  these  results  are  unaffected  by 
the  T v2  value,  since  it  makes  no  difference  on  the  Time  MOE  (it  only  affects  Ev). 
Time  is  consistently  more  favorable  for  larger  Na  values;  even  the  variance  gets 
smaller  as  Na  grows.  So,  for  time  critical  missions,  sending  as  many  type-A 
UCAVs  as  possible,  is  always  a  good  strategy,  no  matter  how  small  the 
difference  of  the  two  types  of  UCAVs  (up  to  a  reasonable  point)  or  how  small  the 
difference  in  the  operational  values  of  the  targets. 


Na 

Mean(Ev) 

StDev(Ev) 

Min(Ev) 

Max(Ev) 

Range(Ev) 

0 

3.869 

1.305 

1.772 

7.170 

5.398 

1 

3.897 

1.287 

1.918 

7.111 

5.193 

2 

3.924 

1.271 

2.057 

7.050 

4.993 

3 

3.949 

1.258 

2.092 

6.988 

4.896 

4 

3.972 

1.249 

2.098 

6.924 

4.826 

5 

3.993 

1.241 

2.082 

6.858 

4.777 

6 

4.013 

1.237 

2.065 

6.791 

4.726 

7 

4.031 

1.235 

2.048 

6.723 

4.675 

8 

4.047 

1.235 

2.031 

6.653 

4.622 

9 

4.061 

1.239 

2.013 

6.667 

4.653 

10 

4.074 

1.245 

1.995 

6.799 

4.804 

11 

4.085 

1.254 

1.977 

6.929 

4.952 

12 

4.095 

1.266 

1.958 

7.056 

5.098 

13 

4.103 

1.280 

1.939 

7.179 

5.240 

14 

4.109 

1.298 

1.920 

7.299 

5.380 

15 

4.114 

1.317 

1.900 

7.416 

5.516 

16 

4.118 

1.340 

1.879 

7.529 

5.650 

Table  9.  Statistics  on  the  Ev  MOE  for  the  different  Na  values  for  Tv2  =  0.5. 
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Na 

Mean(Ev) 

StDev(Ev) 

Min(Ev) 

Max(Ev) 

Range(Ev) 

0 

5.343 

1.686 

2.286 

8.726 

6.440 

1 

5.357 

1.659 

2.447 

8.632 

6.185 

2 

5.369 

1.634 

2.601 

8.544 

5.942 

3 

5.379 

1.614 

2.749 

8.575 

5.826 

4 

5.387 

1.596 

2.890 

8.603 

5.713 

5 

5.393 

1.583 

2.937 

8.628 

5.692 

6 

5.396 

1.573 

2.935 

8.651 

5.715 

7 

5.397 

1.567 

2.895 

8.670 

5.775 

8 

5.396 

1.565 

2.855 

8.687 

5.833 

9 

5.393 

1.567 

2.813 

8.701 

5.888 

10 

5.387 

1.573 

2.771 

8.747 

5.976 

11 

5.379 

1.584 

2.728 

8.886 

6.158 

12 

5.369 

1.599 

2.684 

9.023 

6.339 

13 

5.356 

1.619 

2.639 

9.157 

6.518 

14 

5.341 

1.644 

2.593 

9.289 

6.696 

15 

5.324 

1.673 

2.547 

9.419 

6.872 

16 

5.305 

1.707 

2.499 

9.546 

7.047 

Table  10.  Statistics  on  the  Ev  MOE  for  the  different  Na  values  for  Tv2 


Na 

Mean(Time) 

StDev(Time) 

Min(Time) 

Max(Time) 

Range(Time) 

0 

27.953 

27.697 

8.205 

162.267 

154.062 

1 

27.550 

26.162 

8.062 

153.032 

144.970 

2 

27.133 

24.593 

7.916 

143.425 

135.509 

3 

26.702 

23.005 

7.767 

133.490 

125.722 

4 

26.258 

21.415 

7.615 

123.284 

115.670 

5 

25.803 

19.844 

7.459 

112.885 

105.426 

6 

25.339 

18.322 

7.301 

102.384 

95.083 

7 

24.867 

16.880 

7.139 

91.888 

84.748 

8 

24.391 

15.554 

6.975 

81.516 

74.541 

9 

23.912 

14.383 

6.809 

71.395 

64.586 

10 

23.433 

13.404 

6.640 

61.649 

55.009 

11 

22.956 

12.648 

6.468 

54.610 

48.141 

12 

22.481 

12.133 

6.295 

55.150 

48.854 

13 

22.008 

11.862 

6.121 

55.694 

49.573 

14 

21.534 

1 1 .822 

5.944 

56.241 

50.296 

15 

21.054 

11.994 

5.767 

56.790 

51.023 

16 

20.560 

12.358 

5.588 

57.341 

51.753 

Table  1 1 .  Statistics  on  the  Time  MOE  for  the  different  Na  values 
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Na 


Figure  6.  The  average  Time  MOE  values  as  a  function  of  NA 


Next  we  build  three  regression  meta-models  of  the  mathematical 
model.  They  describe  the  Ev  MOE  only  (since  due  to  heteroscedasticity  we 
deem  Time  inappropriate  for  regression).  The  first  regression  only  employs  the 
(linear)  model  factors,  the  second  one  additionally  employs  quadratic  terms,  and 
the  third  one  additionally  employs  all  possible  second  degree  interactions  of  the 
model  factors.  It  is  obvious  that  the  third  meta-model  has  the  most  descriptive 
power,  but  we  explore  the  other  two  cases  because  simplicity  and  fewer  terms 
are  desirable  regression  attributes  (they  aid  the  interpretation)  even  when  the 
descriptive  power  falls  a  little  bit  behind. 


(1 )  Regression  with  linear  terms  only. 

All  13  factors  are  employed  (none  is  eliminated).  The 
resulting  R2  is  about  87%. 


Response  Ev 
Summary  of  Fit 


RSquare 

0.874 

RSquare  Adj 

0.873 

Root  Mean  Square  Error 

0.613 

Mean  of  Response 

4.027 

Observations  (or  Sum  Wgts) 

1683 
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Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

-7.069 

0.199 

-35.50 

<.0001 

Ti 

0.341 

0.011 

32.38 

<.0001 

t2 

0.107 

0.011 

10.09 

<.0001 

t3 

-0.077 

0.002 

-42.85 

<.0001 

qAi 

1.548 

0.101 

15.33 

<.0001 

qB2 

0.591 

0.101 

5.84 

<.0001 

Ta 

2.485 

0.101 

24.59 

<.0001 

Tb 

2.649 

0.101 

26.15 

<.0001 

Pai 

2.894 

0.084 

34.37 

<.0001 

PB2 

1.373 

0.084 

16.31 

<.0001 

Aa 

0.483 

0.063 

7.67 

<.0001 

0a 

-14.807 

2.015 

-7.35 

<.0001 

Tv2 

2.685 

0.037 

73.28 

0.0000 

Na 

0.015 

0.003 

5.07 

<.0001 

Table  12.  Regression  with  linear  terms  only  output 

(2)  Regression  with  quadratic  terms  also 

This  time,  the  squared  factors  are  also  considered  for  the 
regression.  After  the  elimination  process,  we  are  left  with  22  estimates.  The 


corresponding  terms  are: 


Tj  ,Tj  ,T2  ,T2  T3  ,Tv2  ,qA1  ,qB2  ,qB2  ,rA  ,rA , 

rB  Tb  ’P  A1  ’P  A1  ’P B2  ’P  B2  ’^A  A  A  A  1 


The  R2  is  increased 


by  about  2%  compared  to  the  R2  of  the  linear  regression  above.  Thus,  this 
quadratic  regression  is  deemed  inefficient  compared  to  the  linear  one  above. 


Response  Ev 
Summary  of  Fit 


RSquare 

0.896 

RSquare  Adj 

0.894 

Root  Mean  Square  Error 

0.559 

Mean  of  Response 

4.027 

Observations  (or  Sum  Wgts) 

1683 

Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

-7.380 

0.194 

-38.03 

<.0001 

Ti 

0.344 

0.009 

35.71 

<.0001 

(Ti-5)2 

0.052 

0.012 

4.37 

<.0001 

t2 

0.109 

0.009 

11.27 

<.0001 

(T2-5)2 

0.029 

0.013 

2.26 

0.0240 

t3 

-0.077 

0.002 

-47.00 

<.0001 

(T3-14.0606)2 

0.002 

0.0005 

4.02 

<.0001 

qAi 

1.535 

0.092 

16.68 

<.0001 

qe2 

0.587 

0.092 

6.36 

<.0001 

(qB2-0.7503)2 

4.435 

1.306 

3.40 

0.0007 

Ta 

2.479 

0.092 

26.91 

<.0001 
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Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

(rA-0.7503)2 

5.358 

1.026 

5.22 

<.0001 

|"b 

2.656 

0.092 

28.75 

<.0001 

(re-0.7503)2 

12.969 

1.368 

9.48 

<.0001 

Pai 

2.908 

0.077 

37.91 

<.0001 

(Pai-0.70061)2 

-9.674 

0.976 

-9.91 

<.0001 

PB2 

1.362 

0.077 

17.76 

<.0001 

(pB2-0. 70061  )2 

-4.939 

0.908 

-5.44 

<.0001 

Aa 

0.481 

0.057 

8.38 

<.0001 

0a 

-14.716 

1.836 

-8.01 

<.0001 

(Oa-0.0175)2 

-1023.385 

564.542 

-1.81 

0.0700 

Tv2 

2.685 

0.033 

80.43 

0.0000 

Na 

0.015 

0.003 

5.56 

<.0001 

Table  13.  Regression  with  linear  and  quadratic  terms  output 


(3)  Regression  with  all  linear,  quadratic,  and  second- 
degree  interaction  terms 

This  regression  considers  all  the  factors,  their  squares,  and 
their  possible  second  degree  interactions.  After  the  elimination  process  the 
terms  left  in  the  regression  are  52  which  render  the  use  of  this  regression 
unreasonable,  even  though  the  R2  has  climbed  up  to  99%. 


c.  Results 

By  employing  this  narrow  experimental  design  setup,  we  reached 

the  following  results: 

•  A  larger  NA  is  always  better  for  the  military  value  of  the  operation,  but  it  is 
only  worth  the  extra  cost  when  the  differences  between  the  two  UCAV 
types  are  substantial  and  the  operational  values  of  the  two  types  of  targets 
are  quite  apart. 

•  For  time  critical  missions,  sending  as  many  type-A  UCAVs  as  possible,  is 
always  a  good  strategy,  no  matter  how  small  the  difference  of  the  two 
types  of  UCAVs  (up  to  a  reasonable  point)  or  how  small  the  difference  in 
the  operational  values  of  the  targets. 

•  The  regression  displayed  in  Table  12  (linear  terms  only),  is  a  good  quick 
substitute  for  the  model,  since  it  accounts  for  87%  of  the  variability, 
without  having  to  run  the  model. 
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F.  BASE  CASE  AND  SENSITIVITY  ANALYSIS 

In  this  chapter  we  define  a  base  case  for  our  scenario  and  then  explore 
the  effect  of  each  parameter  on  the  outcome,  when  the  rest  of  the  parameters 
remain  fixed  at  their  base-case  values.  This  type  of  analysis,  which  may  not 
reveal  possible  interactions  among  factors,  complements  the  DOE  setup 
described  in  Sections  D  and  E.  It  might  reveal  potential  relations  that  were 
concealed  in  the  DOE  analysis,  due  to  averaging  and  canceling-out  phenomena. 

1.  Base  Case 

In  the  previous  section  it  was  shown  that  we  can  get  clearer  and  more 
consistent  results  if  we  reduce  the  dimensionality  and  variability  of  the  DOE,  by 
correlating  the  six  factors  qA2,  qBi,  Pa2,  Pbi,  Ab,  and  0B,  to  six  corresponding 
factors  qB2,  qAi,  Pb2,  Pai,  Aa,  and  0A  respectively.  Therefore,  for  the  sensitivity 
analysis  setup,  the  varying  factors  are  twelve  (the  other  six  factors  are 
dependent).  This  time,  the  independence  assumption  of  the  varying  factors  is 
not  required,  as  was  the  case  with  the  NOLH  DOEs  previously  discussed. 

a.  Decision  Variable 

As  in  the  analyses  in  the  previous  sections,  we  only  have  one 
decision  variable,  NA,  which  determines  the  UCAV  mix.  The  total  number  of 
UCAVs  is  fixed  at  16,  and  in  the  base  case,  NA  =  8. 

b.  Scenario  and  Design  Factors 

(1)  Scenario  factors 

•  Ti  =5 

•  T2  =  5 

•  T3=10 

•  Tvi  =  1 

•  Tv2  =  0-5 


(2)  UCAV-design  parameters 


_  4b2 

q“”  71 

qB1  -  — 

C|b2  =  0.8 
rA  =  0.8 
rB  =  0.8 
Pai  =  0.9 

„  —  Pb2 

_  -  Pa, 

PB,  -  — 

Pb2  =  0.9 
Aa=  1 

Ab  =  — 
1.2 

0a  =  0.01 

eB  =  — 

1.2 


[correlated  with  qB2] 
[correlated  with  qAi] 


[correlated  with  pB2] 
[correlated  with  pAi] 


[correlated  with  Aa] 


[correlated  with  0A] 


c.  Results  of  the  Base  Case 

Running  the  model  at  the  base  case  parameter  values,  we  get 
results  for  the  two  MOE  values  as  functions  of  NA,  which  are  displayed  in  Table 
14.  These  results  are  graphically  depicted  in  Figure  7  and  Figure  8  for  Ev  and 
Time  respectively.  We  can  see  that  the  optimal  mix  of  UCAVs  for  the  base  case 
is  NA  =  14  and  NB  =  3,  and  that  the  shortest  Time  value  occurs  at  NA  =  16,  so  it 
appears  that  a  larger  type-A  UCAV  presence  provides  for  better  effectiveness 
and  time-efficiency. 
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Na 

Ev 

Time 

0 

3.921 

13.940 

1 

4.110 

13.738 

2 

4.284 

13.531 

3 

4.444 

13.322 

4 

4.589 

13.112 

5 

4.718 

12.906 

6 

4.831 

12.706 

7 

4.928 

12.517 

8 

5.009 

12.340 

9 

5.074 

12.179 

10 

5.123 

12.036 

11 

5.156 

11.912 

12 

5.174 

11.810 

13 

5.177 

11.730 

14 

5.166 

11.671 

15 

5.142 

11.634 

16 

5.104 

11.616 

Table  14.  Both  MOE  values  for  the  different  Na  values. 
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Figure  8.  The  average  Time  MOE  values  as  a  function  of  NA 


2.  Sensitivity  Analysis 

a.  Course  of  Analysis 

Each  one  of  the  twelve  varying  factors  -  T i ,  T2,  T3,  TV2,  qAi,  qB2,  rA, 
rB,  Pai,  Pb2,  Aa,  0a  -  consecutively  varies  within  its  range,  everything  else  kept 
constant.  Values  of  the  MOEs  are  calculated  for  all  17  NA  values  during  this 
process,  so  that  plots  of  the  MOEs  as  functions  of  the  decision  factor  can  be 
plotted  for  the  different  levels  of  the  examined  scenario  or  design  factor. 

Additionally,  we  explore  the  effect  on  Ev  of  the  T1/T2  ratio  and 
handicap  ratios  on  Ev. 

b.  Sensitivity  Analysis  on  Ti 

T-i  takes  on  values  in  the  set  {3,  5,  7}.  The  expected  value  of  killed 
targets  consistently  increases  with  T 1 ,  at  every  NA  value.  The  increase  rate  is 
higher  for  larger  Ti  values.  The  expected  operation  time  decreases  as  the 
number  of  type-1  targets  increases.  This  might  seem  counterintuitive,  but  in  fact 
the  more  targets  there  are  in  the  area  of  interest,  the  higher  the  acquisition  rate 
of  the  UCAVs  is,  and  this  leads  to  a  decrease  in  the  expected  operation  time. 
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Note  that  the  combat  situation  is  considered  terminated  when  there  are  no 
UCAVs  left,  but  there  can  still  be  alive  targets  left  in  the  battlefield. 

The  overall  conclusion  here  is  that  the  operational  benefit  increases 
with  the  number  of  type-A  UCAVs,  but  that  this  effect  is  even  more  pronounced 
as  Ti  increases.  So,  given  a  high  Ti  value,  the  use  of  more  type-A  UCAVs 
becomes  more  imperative  versus  the  use  of  type-B  ones.  Of  course,  time  critical 
missions  dictate  the  use  of  more  type-A  UCAVs,  no  matter  how  many  type-1 
targets  there  are. 


Figure  9.  Ev  as  a  function  of  Na  for  different  Ti  parameter  values 
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T1  =3  — ■— T1=5  — ± — T1  =7 


Figure  10.  Time  as  a  function  of  Na  for  different  Ti  parameter  values 

c.  Sensitivity  Analysis  on  T2 

T2  takes  on  values  in  the  set  {3,  5,  7}.  The  plots  of  Ev  and  Time  as 
a  function  of  T2  are  similar  to  the  previous  plots  forTi,  and  the  same  conclusions 
naturally  hold.  The  fact  that  the  total  Ev  value  is  smaller  in  the  present  case  is 
attributed  to  the  Tvi  base  case  value  being  twice  that  of  Tv2,  and  so  Ti  has  a 
greater  effect  on  the  Ev  maximum  value  than  T2  does.  But,  T2  seems  to  have  a 
greater  effect  on  the  Ev  increase  rate.  The  conclusion  here  is  that  a  high  Na 
value  is  always  more  desirable,  even  for  small  T2  values. 
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Figure  11.  Ev  as  a  function  of  Na  for  different  T2  parameter  values 


Figure  12.  Time  as  a  function  of  Na  for  different  T2  parameter  values 
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d.  Sensitivity  Analysis  on  T3 

T3  takes  on  values  in  the  set  {0,  14,  28}.  As  the  number  of  active 
NVTs  increases,  more  UCAVs  are  attrite  without  adding  any  operational  value  to 
the  combat  mission.  The  Time  MOE  is  also  negatively  affected  by  a  larger  T3 
number.  Nevertheless,  T3  is  an  environmental  noise  factor,  on  which  we  can 
exercise  no  control.  The  conclusion  here  is  that  the  mission  mix  is  not  a  crucial 
factor  for  the  expected  military  value,  but  when  time  is  of  essence,  a  larger 
number  of  type-A  targets  gives  a  considerably  smaller  length  of  operation. 


Nu m  be r  of  type-A  UCAVs 


■4—  T3=0  — ■— T3=14  — * —  T3=28 


Figure  13.  Ev  as  a  function  of  NA  for  different  T3  parameter  values 


4—  T3=0  — ■—  T3=1 4  —A—  T3=28 


Figure  14.  Time  as  a  function  of  NA  for  different  T3  parameter  values 
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e.  Sensitivity  Analysis  on  TV2 

TV2  takes  on  values  in  the  set  {0,  0.5,  1}.  Obviously,  TV2  does  not 
affect  Time,  but  only  affects  Ev.  The  conclusion  is  that  for  Tv2  values  comparable 
or  equal  to  Tvi  values,  the  mission  mix  is  irrelevant,  but  for  comparatively  small 
T v2  values,  a  larger  number  of  type-A  UCAVs  generates  a  higher  operational 
value. 


Figure  1 5.  Ev  as  a  function  of  Na  for  different  Tv2  parameter  values 


Figure  16.  Time  as  a  function  of  Na  for  different  Tv2  parameter  values 
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f.  Sensitivity  Analysis  on  qM 

qAi  takes  on  values  in  the  set  {0,  0.5,  1}.  What  we  see  here  is  that 
the  unrealistic  qAi  value  of  0,  accounts  for  totally  unfavorable  MOE  results. 
Nevertheless,  cutting  the  qAi  value  in  half  (from  its  maximum  possible  value)  has 
only  minimal  effects  on  both  MOE  values.  A  value  of  0.5  is  already  too  low  for 
realistic  situations;  therefore  the  conclusion  here  is  that  improving  the  recognition 
capability  of  type-A  UCAVs  should  not  be  a  high  priority. 


Figure  17.  Ev  as  a  function  of  NA  for  different  qAi  parameter  values 


Figure  18.  Time  as  a  function  of  NA  for  different  qAi  parameter  values 
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g.  Sensitivity  Analysis  on  qBi 

qB2  takes  on  values  in  the  set  {0,  0.5,  1}.  When  qB2  varies  we 
observe  an  identical  behavior  as  when  qAi  varied.  Thus,  we  should  not  primarily 
spend  our  resources  to  improve  this  capability  of  type-B  UCAVs,  either. 


Figure  19.  Ev  as  a  function  of  Na  for  different  qB2  parameter  values 


Figure  20.  Time  as  a  function  of  NA  for  different  qB2  parameter  values 


h.  Sensitivity  Analysis  on  rA 

rA  takes  on  values  in  the  set  {0,  0.5,  1}.  By  observing  the  following 
two  plots  we  reach  many  conclusions.  Firstly,  if  the  type-A  UCAV  has  perfect 
capability  to  recognize  NVTs,  then  the  more  the  type-A  UCAVs  the  higher  the 
expected  operational  value.  The  expected  operation  time,  though,  explodes.  As 
rA  approaches  0,  smaller  NA  values  give  a  more  favorable  Ev  (and  make  no 
difference  to  the  (small)  expected  operation  time).  If  there  is  no  data  about  the  rA 
value,  then  the  best  strategy  is  to  employ  type-B  UCAVs  only. 


Figure  21 .  Ev  as  a  function  of  NA  for  different  rA  parameter  values 


Figure  22.  Time  as  a  function  of  NA  for  different  rA  parameter  values 
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/'.  Sensitivity  Analysis  on  rB 

rB  takes  on  values  in  the  set  {0,  0.5,  1}.  The  results  here  are 
symmetrical  to  the  results  of  the  previous  case.  If  the  type-B  UCAV  capability  to 
recognize  NVTs  is  close  to  perfect,  then  the  more  the  type-B  UCAVs  the  higher 
the  expected  operational  value.  This,  also,  happens  at  a  high  cost  in  operational 
time.  As  rB  approaches  0,  larger  NA  values  give  a  more  favorable  Ev  (and  make 
no  difference  to  the  (small)  expected  operation  time).  If  there  is  no  data  about 
the  rB  value,  then  the  best  strategy  is  to  employ  type-A  UCAVs  only. 


j.  Sensitivity  Analysis  on  Pai 

Pai  takes  on  values  in  the  set  {0,  0.5,  1}.  For  pAi  =  1 ,  a  larger  NA  is 
desirable.  But  the  smaller  the  pAi  value  gets  the  better  we  are  with  a  smaller  NA. 
Effects  on  Time  are  less  dramatic  than  they  are  on  Ev.  So,  depending  on 
whether  pAi  is  closer  to  1  or  to  0,  the  strategy  differs. 


k.  Sensitivity  Analysis  on  pB2 

pB2  takes  on  values  in  the  set  {0,  0.5,  1}.  The  conclusion  here  is 
that  we  are  always  better  off  with  as  many  type-A  UCAVs  as  possible. 
Nonetheless,  for  pe2  =  1,  the  mission  mix  becomes  less  important. 


Num  ber  of  type-A  UCAV  s 


-  pB2=0  — ■—  pB2=0.5  —A—  pB2=1 


Figure  27.  Ev  as  a  function  of  Na  for  different  Pb2  parameter  values 


Number  of  type-A  UCAVs 


-  pB2=0  — ■ —  pB2=0.5  —a —  pB2=1 


Figure  28.  Time  as  a  function  of  NA  for  different  qB2  parameter  values 


I.  Sensitivity  Analysis  on  Aa 

Aa  takes  on  values  in  the  set  {0.5,  1,  2}.  The  plots  for  this  factor 
show  that  Ev  is  not  significantly  affected  by  Aa,  but  Time  is.  We  conclude  that  it  is 
the  magnitude  of  Aa  (and  therefore  of  Ab  too)  that  determines  the  best  mission 
mix  time-wise.  When  Aa  is  large  enough,  we  should  strive  for  more  type-A 
UCAVs,  otherwise  the  opposite  is  true. 


Figure  29.  Ev  as  a  function  of  NA  for  different  Aa  parameter  values 


Figure  30.  Time  as  a  function  of  Na  for  different  Aa  parameter  values 
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m.  Sensitivity  Analysis  on  6A 

0A  takes  on  values  in  the  set  {0.005,  0.010,  0.020}.  The  effect  of  0A 
on  Ev  is  minimal.  The  same  is  true  for  Time.  Note  that  if  the  operation  time  is  a 
concern,  then  the  best  strategy  remains  to  employ  as  many  type-A  UCAVs  as 
possible,  regardless  of  the  0A  value. 


Figure  31 .  Ev  as  a  function  of  NA  for  different  0A  parameter  values 


Figure  32.  Time  as  a  function  of  NA  for  different  0A  parameter  values 


n.  Additional  Exploration 

In  this  section,  we  explore  the  effect  that  the  Ti/T2  ratio  and  the 
handicap  ratio  have  on  the  Ev  value.  This  is  one  example  of  the  many  uses  of 
our  model.  It  demonstrates  that  by  simultaneously  varying  more  than  one  factor 
(and  more  so  ratios  of  factors)  we  can  gain  additional  insight,  due  to  interactions 
revealed.  This  example  lies  between  the  NOLH  DOE  and  the  strict  one¬ 
dimensional  sensitivity  analysis  approaches. 

The  three  figures  below  expose  the  main  concept:  if  the  ratio  T1/T2 
is  equal  to  1,  then  the  optimal  UCAV  mix  balances  in  the  middle  (i.e.,  NA  =  8). 
When  Ti/T2  is  greater  than  1,  then  the  optimal  mix  tends  to  be  displaced  to  the 
right  (i.e.,  NA  >  8),  and  when  the  ratio  is  smaller  than  1,  the  best  mission  mix  is 
displaced  to  the  left  (i.e.,  NA  <  8).  The  further  the  T1/T2  ratio  is  from  1,  the  further 
the  optimal  NA  value  is  from  the  middle  value  (i.e.,  8).  These  effects  are  more 
dramatic,  for  larger  handicaps,  as  is  displayed  by  the  three  plots  in  each  figure. 
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Number  of  type-A  UCAV's 


G.  CONCLUSIONS 

As  is  often  the  case  with  DoD  exploratory  analysis,  the  model  has  very 
limited  predictive  power.  Indeed,  due  to  a  lack  of  data,  the  model  cannot  be 
empirically  validated.  Rather,  the  model  is  used  in  a  descriptive  mode  to  help  us 
devise  new  ideas  or  assess  the  consequences  of  certain  assumptions.  Potential 
insights  gleaned  from  such  exploration  usually  need  to  be  tested  elsewhere, 
perhaps  by  field  experiments. 

In  this  exploratory  analysis  we  are  primarily  trying  to  identify  the  optimal 
mission  mix,  and  secondarily  the  factors  that  have  a  strong  effect  on  the  MOE 
values,  the  directions  of  those  effects,  and  which,  if  any,  factors  interact. 

By  employing  the  broad  experimental  design  setup,  we  observe  that  NA 
(mission  mix)  does  not  significantly  affect  Ev  (expected  military  value).  This 
observation  is  due  to  the  balancing  effect  of  the  factor  values.  Although  there  is 
some  effect  on  the  other  MOE,  Time,  it  does  not  seem  to  be  that  significant 
either.  Nevertheless,  for  a  more  robust  Time  outcome,  absent  hard  data,  it  is 
better  to  employ  a  balanced  mix  of  UCAVs  of  different  types,  instead  of  a  biased 
one  where  most  of  the  UCAVs  are  of  the  same  type. 

By  employing  the  narrow  experimental  design  setup,  we  decreased  the 
overall  design  noise,  and  we  reached  the  following  results: 

•  A  larger  value  of  NA  is  always  better  for  the  military  value  of  the  operation, 
but  it  is  only  worth  the  extra  cost  (note  that  a  type-A  UCAV  is  more  costly 
than  a  type-B  UCAV)  when  the  differences  in  design  characteristics  (like 
kill  probability,  detection  rate,  etc)  between  the  two  UCAV  types  are 
substantial  and  the  operational  values  of  the  two  types  of  targets  are  quite 
apart. 

•  For  time  critical  missions,  sending  as  many  type-A  UCAVs  as  possible,  is 
always  a  good  strategy,  no  matter  how  small  the  difference  between  the 
two  types  of  UCAVs  (note  that  if  there  are  no  differences  then  type-B 
UCAVs  should  be  employed  exclusively  since  they  are  less  expensive)  or 
how  small  the  difference  in  the  operational  values  of  the  targets. 
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By  running  a  sensitivity  analysis  on  our  model,  we  reach  some 

conclusions  pertaining  to  the  individual  parameters  as  follows: 

•  Given  a  high  Ti  value  (number  of  type-1  targets),  the  use  of  more  type-A 
UCAVs  becomes  more  imperative  versus  the  use  of  type-B  ones.  Of 
course,  time  critical  missions  dictate  the  use  of  more  type-A  UCAVs,  no 
matter  how  many  type-1  targets  there  are. 

•  A  higher  NA  value  is  always  more  desirable,  even  for  small  T2  values 
(number  of  type-2  targets). 

•  For  TV2  values  (type-2  target  military  value)  comparable  to  TVi  values 
(type-1  target  military  value),  the  mission  mix  is  irrelevant,  but  for 
comparatively  small  Tv2  values,  a  larger  number  of  type-A  UCAVs 
generates  a  higher  operational  value. 

•  Improving  the  recognition  capabilities  (which  are  typically  good  enough 
already)  of  UCAVs  (of  either  type)  should  not  be  on  a  high  priority  (i.e. ,  we 
should  allocate  available  resources  into  improving  other  aspects  first). 

•  If  there  is  no  data  about  the  recognition  capabilities  of  a  type-A  UCAV  (rA), 
then  the  best  strategy  is  to  employ  type-B  UCAVs  only.  Otherwise,  if  the 
type-A  UCAV  capability  to  recognize  NVTs  is  close  to  perfect,  then  the 
more  the  type-A  UCAVs  the  higher  the  expected  operational  value.  In  this 
case,  though,  the  expected  operation  time  gets  long.  As  rA  approaches  0, 
smaller  NA  values  give  a  more  favorable  Ev  (and  make  no  difference  to  the 
(small)  expected  operation  time). 

•  The  conclusions  for  re  (recognition  capability  of  type-B  UCAV)  are 
symmetrical  to  the  conclusions  for  rA  stated  above. 

•  For  high  pA1  (kill  probability  of  type-A  UCAV  against  a  type-1  target)  values 
(i.e.,  close  to  1),  a  larger  NA  is  desirable.  The  smaller  the  pAi  value  gets 
the  better  we  are  with  a  smaller  NA.  Effects  on  Time  are  less  dramatic 
than  they  are  on  Ev.  So,  depending  on  whether  pA1  is  closer  to  1  or  to  0, 
the  strategy  differs. 
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•  No  matter  what  the  value  of  pB2  (kill  probability  of  type-B  UCAV  against  a 
type-2  target)  is,  we  are  always  better  off  with  as  many  type-A  UCAVs  as 
possible.  Nonetheless,  for  very  high  pB2  values  the  mission  mix  becomes 
less  important.  Note  that,  as  it  is  implied  by  the  experimental  design,  the 
pB2  value  is  correlated  (i.e.,  not  far  from)  to  the  pA2  value,  for  this 
conclusion  to  hold. 

•  Ev  is  not  significantly  affected  by  Aa  (type-A  UCAV  detection  rate),  but 
Time  is,  and  it  is  the  relationship  between  Aa  and  AB  (type-B  UCAV 
detection  rate)  that  determines  the  best  mission  mix  time-wise.  When  Aa 
is  greater  than  AB,  we  should  strive  for  more  type-A  UCAVs  and  when  AB  is 
greater  than  Aa  the  opposite  is  true. 

•  The  effect  of  0A  (type-A  UCAV  failure  rate)  on  Ev  is  minimal.  The  same  is 
true  for  Time.  Of  course  this  is  due  to  the  large  —  ratio  (by  the  UCAV 

design). 
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APPENDIX.  MATLAB  CODE  FOR  THE  BASIC  MODEL 


This  Appendix  contains  the  Matlab  code  for  the  basic  model.  By  adopting 
this  code  and  slightly  modifying  it,  we  can  also  implement  the  extension  of  the 
basic  model  presented  in  Chapter  IV,  or  even  other  potential  extensions. 


format  compact 

format  short 

A  =  xlsread('DOEinput.xls'); 
[row,  col]  =  size(A); 

N  =  16;  %hard-wired  value 

u  =  zeros(2); 
u(1,1)  =  0; 
u(1,2)  =  0; 
u(2,1)  =  0; 
u(2,2)  =  0; 

for  j  =  1:row 

T1  =  A(j,1);  %input 
T2  =  A(j ,2);  %input 
T3  =  A(j,3);  %input 

Tvalue  =  zeros(2,1); 
Tvalue(l)  =  A(j,4);  %input 
Tvalue(2)  =  A(j,5);  %input 

q  =  zeros(2); 
q(1,1)  =  A(j,6);  %input 
q(2,1)  =  A(j,7);  %input 
q(2,2)  =  A(j,8);  %input 
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q(1 ,2)  =  A(j,9);  %input 


r  =  zeros(2,1); 
r(1)  =  A(j,10);  %input 
r(2)  =  A(j,11);  %input 

p  =  zeros(2); 
p(1,1)  =  A(j,12);  %input 
p(2,1)  =  A(j,13);  %input 
p(2,2)  =  A(j,14);  %input 
p(1 ,2)  =  A(j,15);  %input 

lamda  =  zeros(2,1); 
lamda(l)  =  A(j,  1 6);  %input 
lamda(2)  =  A(j,17);  %input 

theta  =  zeros(2,1); 
theta(l)  =  A(j,18);  %input 
theta(2)  =  A(j,19);  %input 

N1  =  A(j ,20);  %input 
N2  =  N-N1;  %derived 

nl  =  0; 
n2  =  0; 
tl  =  0; 
t2  =  0; 


numFeas  =  0;  %counts  total  number  of  feasible  states 
numAbs  =  0;  %counts  total  number  of  absorbing  states 

rn  =  (Nl  +  1)*(N2  +  1)*(T1  +  1)*(T2  +  1); 

state  =  zeros(rn,  6); 

for  nl  =  0:N1 
for  n2  =  0:N2 
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for  tl  =  0:T  1 
for  t2  =  0:T2 

if(T  1 +T2-N 1  -N2<=t1  +t2-n  1  -n2  &  n1>=0  &  n2>=0  &  t1>=0  &  t2>=0) 
numFeas  =  numFeas  +  1; 
state(numFeas,  1)  =  numFeas; 
state(numFeas,  2)  =  nl; 
state(numFeas,  3)  =  n2; 
state(numFeas,  4)  =  tl; 
state(numFeas,  5)  =  t2; 
state(numFeas,  6)  =  0; 
if((n1  +n2)  ==  0  ) 

numAbs  =  numAbs  +  1; 

state(numFeas,  6)  =  1;  %this  flags  an  absorbing  state 
end 
end 
end 
end 
end 
end 

stateAbs  =  zeros(numAbs,  6);  %temp  storage  of  absorbing  states 
numTrans  =  numFeas  -  numAbs;  %number  of  transient  states 
stateTrans  =  zeros(numTrans,  6);  %temp  storage  of  transient  states 
nextAbs  =  1;  %counter  for  the  next  available  line  of  'stateAbs' 
nextTrans  =  1;  %counter  for  the  next  available  line  of  'stateTrans' 

for  ix  =  1  :numFeas 
if(  state(ix,6)  ==  1  ) 

stateAbs(nextAbs,  :)  =  state(ix,  :); 

stateAbs(nextAbs,  1)  =  nextAbs;  %allowing  for  consecutive  indices 
nextAbs  =  nextAbs  +  1 ; 
else 

stateTrans(nextTrans,  :)  =  state(ix,  :); 
stateTrans(nextTrans,  1)  =  numAbs  +  nextTrans; 
nextTrans  =  nextTrans  +  1; 
end 
end 
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state  =  [stateAbs  ;  stateTrans]; 


state  =  state(1  :numFeas,  :); 

P  =  zeros(numFeas); 

for  ix  =  1  :numFeas 
for  iy  =  1:numFeas 

%Case  1:  from  (nl,  n2,  tl,  t2)  to  (n  1  - 1 ,  n2,  tl  - 1,  t2)  state 
if(  state(ix,2)-1  ==  state(iy,2)  &  state(ix,3)  ==  state(iy,3) ... 

&  state(ix,4)-1  ==  state(iy,4)  &  state(ix,5)  ==  state(iy,5) ) 
P(ix,iy)  =  1; 

%Case  2:  from  (nl,  n2,  tl,  t2)  to  (nl  -  1,  n2,  tl,  t2)  state 
elseif  ( (state(ix,2)-1  ==  state(iy,2))  &  (state(ix,3)  ==  ... 
state(iy,3))  &  (state(ix,4)  ==  state(iy,4))  &  ... 

(state(ix,5)  ==  state(iy,5)) ) 

P(ix,iy)  =  2; 

%Case  3:  from  (nl,  n2,  tl,  t2)  to  (nl,  n2  - 1,  tl,  t2)  state 
elseif  ( (state(ix,2)  ==  state(iy,2))  &  (state(ix,3)-1  ==  ... 
state(iy,3))  &  (state(ix,4)  ==  state(iy,4))  &  ... 

(state(ix,5)  ==  state(iy,5)) ) 

P(ix,iy)  =  3; 

%Case  4:  from  (nl,  n2,  tl,  t2)  to  (nl  -  1,  n2,  tl,  t2  -  1)  state 
elseif  ( (state(ix,2)-1  ==  state(iy,2))  &  (state(ix,3)  ==  ... 
state(iy,3))  &  (state(ix,4)  ==  state(iy,4))  &  ... 

(state(ix,5)-1  ==  state(iy,5)) ) 

P(ix,iy)  =  4; 

%Case  5:  from  (nl,  n2,  tl,  t2)  to  (nl,  n2  - 1,  tl  - 1,  t2)  state 
elseif  ( (state(ix,2)  ==  state(iy,2))  &  (state(ix,3)-1  ==  ... 
state(iy,3))  &  (state(ix,4)-1  ==  state(iy,4))  &  ... 

(state(ix,5)  ==  state(iy,5)) ) 

P(ix,iy)  =  5; 

%Case  6:  from  (nl,  n2,  tl,  t2)  to  (nl,  n2  - 1,  tl,  t2  - 1)  state 
elseif  ( (state(ix,2)  ==  state(iy,2))  &  (state(ix,3)-1  ==  ... 
state(iy,3))  &  (state(ix,4)  ==  state(iy,4))  &  ... 

(state(ix,5)-1  —  state(iy,5)) ) 

P(ix,iy)  =  6; 
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%Case  7:  from  (n  1 ,  n2,  tl,  t2)  to  (n  1 ,  n2,  tl,  t2)  state  (the  same) 
elseif  ( (state(ix,2)  ==  state(iy,2))  &  (state(ix,3)  ==  ... 
state(iy,3))  &  (state(ix,4)  ==  state(iy,4))  &  ... 

(state(ix,5)  ==  state(iy,5)) ) 

P(ix,iy)  =  7; 
else 

%None  of  the  7  cases;  meaning  that  no  transition  happens 
P(ix,iy)  =  0; 
end 
end 
end 

[x  1 ,  y  1  ]  =  find(P  ==  1); 

[x2,y2]  =  find(P  ==  2); 

[x3,y3]  =  find(P  ==  3); 

[x4,y4]  =  find(P  ==  4); 

[x5,y5]  =  find(P  ==  5); 

[x6,y6]  =  find(P  ==  6); 

[x7,y7]  =  find(P  ==  7); 

%Case  1 

lenl  =  length(xl); 

for  ix  =  1  :len1 

P(x1  (ix),y1  (ix))  =  lamda(1  )*p(1 ,1  )*state(x1  (ix),2)*state(x1  (ix),4)/ .. 
((Iamda(1)+theta(1))*state(x1(ix),2)  +  (lamda(2)+theta(2)) ... 
*state(x1  (ix),3))/(T1+T2+T3)*(q(1 ,1  )+u(1 ,2)*(1-q(1 ,1 ))); 
end 

%Case  2 
Ien2  =  Iength(x2); 
for  ix  =  1  :len2 

P(x2(ix),y2(ix))  =  state(x2(ix),2)*theta(1)/((lamda(1)+theta(1))  ... 
*state(x2(ix),2)  +  (Iamda(2)+theta(2))*state(x2(ix),3))  +  ... 
state(x2(ix),2)*lamda(1  )/((lamda(1  )+theta(1  ))*state(x2(ix),2)+  .. 
(lamda(2)+theta(2))*state(x2(ix),3))/(T  1 +T2+T3)*  ... 
(state(x2(ix),4)*(q(1 , 1  )+u(1 ,2)*(1-q(1 ,1  )))*(1-p(1 ,1 ))+  ... 
state(x2(ix),5)*(q(1 ,2)+u(1 ,1  )*(1-q(1 ,2)))*(1-p(1 ,2))+  ... 

(T 1  +T2-state(x2(ix),4)-state(x2(ix),5)+T3)*(1-r(1 ))); 
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end 

%Case  3 
Ien3  =  Iength(x3); 
for  ix  =  1  :len3 

P(x3(ix),y3(ix))  =  state(x3(ix),3)*theta(2)/((lamda(1)+theta(1))  ... 
*state(x3(ix),2)  +  (Iamda(2)+theta(2))*state(x3(ix),3))  +  ... 
state(x3(ix),3)*lamda(2)/((lamda(1)+theta(1))*state(x3(ix),2)+ 
(lamda(2)+theta(2))*state(x3(ix),3))/(T  1 +T2+T3)*  ... 
(state(x3(ix),4)*(q(2,1)+u(2,2)*(1-q(2,1)))*(1-p(2,1))+  ... 
state(x3(ix),5)*(q(2,2)+u(2,1)*(1-q(2,2)))*(1-p(2,2))+  ... 
(T1+T2-state(x3(ix),4)-state(x3(ix),5)+T3)*(1-r(2))); 
end 

%Case  4 
Ien4  =  Iength(x4); 
for  ix  =  1  :len4 

P(x4(ix),y4(ix))  =  lamda(1  )*p(1 ,2)*state(x4(ix),2)*state(x4(ix),5)/ 
((Iamda(1)+theta(1))*state(x4(ix),2)  +  (lamda(2)+theta(2)) ... 
*state(x4(ix),3))/(T1+T2+T3)*(q(1 ,2)+u(1 ,1)*(1-q(1 ,2))); 
end 

%Case  5 
Ien5  =  Iength(x5); 
for  ix  =  1  :len5 

P(x5(ix),y5(ix))  =  lamda(2)*p(2,1  )*state(x5(ix),3)*state(x5(ix),4)/ 
((Iamda(1)+theta(1))*state(x5(ix),2)  +  (lamda(2)+theta(2)) ... 
*state(x5(ix),3))/(T1+T2+T3)*(q(2,1)+u(2,2)*(1-q(2,1))); 
end 

%Case  6 
Ien6  =  Iength(x6); 
for  ix  =  1  :len6 

P(x6(ix),y6(ix))  =  Iamda(2)*p(2,2)*state(x6(ix),3)*state(x6(ix),5)/ 
((Iamda(1)+theta(1))*state(x6(ix),2)  +  (lamda(2)+theta(2)) ... 
*state(x6(ix),3))/(T1+T2+T3)*(q(2,2)+u(2,1)*(1-q(2,2))); 
end 

%Case  7 
Ien7  =  Iength(x7); 
for  ix  =  1  :len7 

%Distinguishing  between  absorbing  and  non-absorbing  states 
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if  (state(x7(ix),2)  ==  0  &  state(x7(ix),3)  ==  0) 

P(x7(ix),y7(ix))  =  1; 
else 

P(x7(ix),y7(ix))  =  Iamda(1)*state(x7(ix),2)/ ... 

((Iamda(1)+theta(1))*state(x7(ix),2)  +  (lamda(2)+theta(2))  * ... 
state(x7(ix),3))/(T  1  +T2+T3)*(state(x7(ix),4)* ... 

(1-u(1 ,2))*(1-q(1 ,1))+state(x7(ix),5)*(1-u(1 ,1))* ... 

(1-q(1 ,2))+(T1+T2-state(x7(ix),4)-state(x7(ix),5)+T3)*r(1))  ... 

+  Iamda(2)*state(x7(ix),3)/ ... 

((Iamda(1)+theta(1))*state(x7(ix),2)  +  ... 

(lamda(2)+theta(2))*state(x7(ix),3))/(T1+T2+T3)*(state(x7(ix),4)*... 
(1-u(2,2))*(1-q(2,1))+state(x7(ix),5)*(1-u(2,1))*  ... 
(1-q(2,2))+(T1+T2-state(x7(ix),4)-state(x7(ix),5)+T3)*r(2)); 
end 
end 

numl  =  0;  %stores  the  number  of  lines  that  have  errors 
disp('Checking  Transition  Matrix  for  Integrity  and  Errors') 
for  ix  =  1  :numFeas 

if  (abs(sum(P(ix,:)))  -  1  >=  0.0001) 
disp(['Error  in  Transition  Matrix,  line  '  num2str(ix)]) 
numl  =  numl  +  1 ; 
end 
end 

disp(['Number  of  lines  containing  errors: '  num2str(num1 )]) 

R  =  P(numAbs+1:numFeas,  1:numAbs); 

Q  =  P(numAbs+1  :numFeas,  numAbs+1:numFeas); 

I  =  eye(size(Q)); 
l_Q  =  I  -  Q; 
l_Q_inv  =  inv(l_Q); 
l_Q_inv_R  =  l_Q_inv  *  R; 

num2  =  0;  %stores  the  number  of  lines  that  have  errors 
disp(' ')  %insert  an  empty  line  on  the  screen 
disp('Checking  Trans-to-Absorb  Matrix  for  Integrity  and  Errors') 
for  ix  =  1  :numTrans 
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if  (abs(sum(l_QJnv_R(ix,:)))  - 1  >=  0.0001 ) 
disp(['Error  in  Trans-to-Absorb  Matrix,  line  '  num2str(ix)]) 
num2  =  num2  +  1 ; 
end 
end 

disp(['Number  of  lines  containing  errors: '  num2str(num2)]) 

initialState  =  0; 

for  ix  =  numAbs+1  :numFeas 

if(  state(ix,2)  +  state(ix,3)  +  state(ix,4)  +  state(ix,5)  ==  ... 

N1  +  N2  +  T1  +T2) 

initialState  =  ix  -  numAbs; 
break; 
end 
end 

expNumKilled_T1  =  0; 
for  iy  =  1  :numAbs 

expNumKilled_T1  =  expNumKilled_T1  ... 

+  l_Q_inv_R(initialState,  iy)*(  T1  -  state(iy,4) ); 
end 

expNumKilled_T2  =  0; 
for  iy  =  1  :numAbs 

expNumKilled_T2  =  expNumKilled_T2  ... 

+  l_QJnv_R(initialState,  iy)  *  (  T2  -  state(iy,5) ); 
end 

Ev  =  expNumKilled_T  1  *  Tvalue(1 )  +  expNumKilled_T2  *  Tvalue(2); 


Time  =  0; 

for  k  =  1  :numTrans 

Time  =  Time  +  l_Q_inv(initialState,k)  / ... 

(state(numAbs+k,2)*(lamda(1)+theta(1))+state(numAbs+k,3)* ... 
(Iamda(2)+theta(2))); 
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A(j,  col  +  1)  =  Ev; 
A(j,  col  +  4)  =  Time; 


end 

save  129x17_Model_A_NOLH_output.xls  A  -ascii; 

disp(' ')  %insert  an  empty  line  on  the  screen 
if(  numl  +  num2  ==  0  ) 
disp('Script  successfully  completed.') 
disp('Check  workspace  for  more  results.') 
else 

disp('There  are  flaws  in  the  code.  Debugging  needed.') 
end 
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